# Connect - Full Markdown Export > This file contains all Connect documentation pages in markdown format for AI agent consumption. > Generated from 413 pages on 2026-10-15T16:29:34.813Z > Component: connect | Version: > Site: https://docs.redpanda.com ## About This Export This export includes the **latest version** () of the Connect documentation. ### AI-Friendly Documentation Formats We provide multiple formats optimized for AI consumption: - **https://docs.redpanda.com/llms.txt**: Curated overview of all Redpanda documentation - **https://docs.redpanda.com/llms-full.txt**: Complete documentation export with all components - **https://docs.redpanda.com/connect-full.txt**: This file - Connect documentation only - **Individual markdown pages**: Each HTML page has a corresponding .md file --- # Page 1: Components Catalog **URL**: https://docs.redpanda.com/connect/components/about.md --- # Components Catalog --- title: Components Catalog latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/about.adoc description: A searchable catalog of available Redpanda Connect components. page-git-created-date: "2024-05-24" page-git-modified-date: "2025-08-08" --- Use the following table to search for available inputs, outputs, and processors. For a description of the support levels, see [Support levels](#support-levels). Type: All Types Selected ▼ Buffer Cache Input Metric Output Processor Rate limit Scanner Tracer Support: All Support Levels Selected ▼ Certified Community Available in Cloud: All Options Selected ▼ Yes No Enterprise License: All Options Selected ▼ Yes No | Name | Connector Type | Support Level | Enterprise Licensed | Available in Cloud | | --- | --- | --- | --- | --- | | memory | Buffer, Cache | Certified | No | Yes | | none | Buffer, Metric, Tracer | Certified | No | Yes | | sqlite | Buffer | Community | No | No | | system_window | Buffer | Certified | No | Yes | | aws_dynamodbAWS DynamoDB Amazon DynamoDB DynamoDB | Cache, Output | Community | No | Yes | | aws_s3AWS S3 Amazon S3 S3 Simple Storage Service | Cache, Input, Output | Certified | No | Yes | | couchbase | Cache, Output, Processor | Community | No | No | | file | Cache, Input, Output | Certified | No | No | | gcp_cloud_storageGCP Cloud Storage Google Cloud Storage GCS | Cache, Input, Output | Certified | No | Yes | | lru | Cache | Community | No | Yes | | memcached | Cache | Community | No | Yes | | mongodbMongo | Cache, Input, Output, Processor | Certified | No | Yes | | multilevel | Cache | Certified | No | Yes | | nats_kvNATS KV | Cache, Input, Output, Processor | Certified | No | Yes | | noop | Cache, Processor | Certified | No | Yes | | redis | Cache, Processor, Rate_limit | Certified | No | Yes | | redpanda | Cache, Input, Output, Tracer | Certified | No | Yes | | ristretto | Cache | Community | No | Yes | | sql | Cache, Output, Processor | Certified: cacheCommunity: output, processor | No | Yes | | ttlru | Cache | Community | No | Yes | | amqp_0_9RabbitMQ AMQP | Input, Output | Certified | No | Yes | | amqp_1RabbitMQ AMQP Apache Qpid | Input, Output | Community | No | No | | aws_cloudwatch_logsAWS CloudWatch Logs Amazon CloudWatch Logs | Input | Community | No | Yes | | aws_dynamodb_cdcAmazon DynamoDB CDC | Input | Certified | Yes | Yes | | aws_kinesisAWS Kinesis Amazon Kinesis Kinesis | Input, Output | Certified | No | Yes | | aws_sqsAWS SQS Amazon SQS SQS Simple Queue Service | Input, Output | Certified | No | Yes | | azure_blob_storageAzure Blob Storage Microsoft Azure Storage | Input, Output | Certified | No | Yes | | azure_cosmosdbMicrosoft Azure Azure | Input, Output, Processor | Certified | No | Yes | | azure_queue_storageAzure Queue Storage Microsoft Azure Queue | Input, Output | Certified | No | Yes | | azure_table_storageAzure Table Storage Microsoft Azure Table | Input, Output | Certified | No | Yes | | batched | Input | Certified | No | Yes | | beanstalkd | Input, Output | Community | No | No | | broker | Input, Output | Certified | No | Yes | | cassandraApache Cassandra | Input, Output | Community | No | No | | cockroachdb_changefeed | Input | Community | No | No | | csvComma-Separated Values | Input, Scanner | Certified | No | Yes | | discord | Input, Output | Community | No | No | | dynamic | Input, Output | Community | No | No | | gateway | Input | Certified | Yes | Yes | | gcp_bigquery_selectGCP BigQuery Google Cloud GCP | Input, Processor | Certified | No | Yes | | gcp_pubsubGCP PubSub Google Cloud Pub/Sub GCP Pub/Sub Google Pub/Sub | Input, Output | Certified | No | Yes | | gcp_spanner_cdcGoogle Cloud GCP | Input | Certified | Yes | Yes | | generate | Input | Certified | No | Yes | | git | Input | Certified | No | Yes | | hdfs | Input, Output | Community | No | No | | http_clientHTTP REST API REST | Input, Output | Certified | No | Yes | | http_serverHTTP REST API REST Gateway | Input, Output | Certified | No | Yes | | inproc | Input, Output | Certified | No | Yes | | jiraAtlassian Jira | Input, Processor | Certified | Yes | Yes (not in GPU pipelines) | | kafkaApache Kafka | Input, Output | Certified | No | Yes | | kafka_franzApache Kafka Kafka | Input, Output | Certified | No | Yes | | microsoft_sql_server_cdc | Input | Certified | Yes | Yes | | mongodb_cdcMongoDB CDC | Input | Certified | Yes | Yes | | mqtt | Input, Output | Certified | No | Yes | | mysql_cdc | Input | Certified | Yes | Yes | | nanomsg | Input, Output | Community | No | No | | natsNATS.io | Input, Output | Certified | No | Yes | | nats_jetstreamNATS JetStream NATS | Input, Output | Certified | No | Yes | | nats_streamNATS Stream NATS Streaming NATS | Input, Output | Community | No | No | | nsq | Input, Output | Community | No | No | | ockam_kafka | Input, Output | Community | No | No | | oracledb_cdcOracle CDC OracleDB CDC Oracle Database CDC | Input | Certified | Yes | Yes | | otlp_grpcOpenTelemetry OTLP OTel gRPC | Input, Output | Certified | Yes | Yes | | otlp_httpOpenTelemetry OTLP OTel | Input, Output | Certified | Yes | Yes | | parquet | Input, Processor | Certified: inputCommunity: processor | No | No | | pg_stream | | Certified | Yes | Yes | | postgres_cdc | Input | Certified | Yes | Yes | | pulsarApache Pulsar | Input, Output | Community | No | No | | read_until | Input | Certified | No | Yes | | redis_listRedis List Redis Lists Redis | Input, Output | Certified | No | Yes | | redis_pubsubRedis PubSub Redis Pub/Sub Redis | Input, Output | Certified | No | Yes | | redis_scanRedis | Input | Certified | No | Yes | | redis_streamsRedis Streams Redis | Input, Output | Certified | No | Yes | | redpanda_common | Input, Output | Certified | Yes | Yes | | redpanda_migrator | Input, Output | Certified | No | Yes | | resource | Input, Output, Processor | Certified | No | Yes | | salesforce | Input | Certified | Yes | Yes | | salesforce_cdcSalesforce Salesforce CDC | Input | Certified | Yes | Yes | | salesforce_graphqlSalesforce Salesforce GraphQL | Input | Certified | Yes | Yes | | sap_hanaSAP HANA | Input | Certified | Yes | No | | schema_registry | Input, Output | Certified | No | Yes | | sequence | Input | Certified | No | Yes | | sftp | Input, Output | Certified | No | Yes | | slack | Input | Certified | Yes | Yes | | slack_usersSlack Users | Input | Certified | Yes | Yes | | socket | Input, Output | Certified | No | No | | socket_server | Input | Certified | No | No | | spicedb_watch | Input | Community | No | Yes | | splunk | Input | Certified | Yes | Yes | | sql_rawSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Input, Output, Processor | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_selectSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Input, Processor | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | stdin | Input | Certified | No | No | | subprocess | Input, Output, Processor | Community | No | No | | tigerbeetle_cdc | Input | Certified | No | No | | timeplus | Input, Output | Community | No | Yes | | twitter_search | Input | Community | No | No | | websocket | Input, Output | Certified | No | Yes | | zmq4 | Input, Output | Community | No | No | | aws_cloudwatch | Metric | Community | No | No | | influxdb | Metric | Community | No | No | | json_api | Metric | Certified | No | No | | logger | Metric | Certified | No | No | | open_telemetry_collectorOpenTelemetry | Metric, Tracer | Certified | Yes | Yes | | prometheus | Metric | Certified | No | Yes | | statsd | Metric | Certified | No | No | | arc | Output | Community | No | Yes | | aws_kinesis_firehoseAWS Kinesis Firehose Amazon Kinesis Firehose Kinesis Firehose | Output | Certified | No | Yes | | aws_snsAWS SNS Amazon SNS SNS Simple Notification Service | Output | Community | No | Yes | | azure_data_lake_gen2Microsoft Azure Azure | Output | Certified | No | Yes | | cache | Output, Processor | Certified | No | Yes | | cyborgdb | Output | Community | No | Yes | | cypher | Output | Community | No | No | | doris_stream_load | Output | Community | No | No | | drop | Output | Certified | No | Yes | | drop_on | Output | Certified | No | Yes | | elasticsearch_v8 | Output | Certified | No | Yes | | elasticsearch_v9 | Output | Community | No | No | | fallback | Output | Certified | No | Yes | | gcp_bigqueryGCP BigQuery Google BigQuery BigQuery | Output | Certified | No | Yes | | gcp_bigquery_write_apiGCP BigQuery | Output | Certified | Yes | Yes | | icebergApache Iceberg Apache Polaris AWS Glue Databricks Unity Catalog | Output | Certified | Yes | Yes | | opensearch | Output | Certified | No | Yes | | pinecone | Output | Certified | No | Yes | | pusher | Output | Community | No | No | | qdrant | Output, Processor | Certified | No | Yes | | questdb | Output | Certified | No | Yes | | redis_hashRedis Hash Redis | Output | Certified | No | Yes | | reject | Output | Certified | No | Yes | | reject_errored | Output | Certified | No | Yes | | retry | Output, Processor | Certified | No | Yes | | salesforce_sinkSalesforce Salesforce Sink | Output | Certified | Yes | Yes | | slack_postSlack Post | Output | Certified | Yes | Yes | | slack_reactionSlack Reaction | Output | Certified | Yes | Yes | | snowflake_putSnowflake | Output | Certified | Yes | Yes | | snowflake_streamingSnowflake Streaming | Output | Certified | Yes | Yes | | splunk_hecSplunk | Output | Certified | Yes | Yes | | sql_insertSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Output, Processor | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | stdout | Output | Certified | No | No | | switch | Output, Processor, Scanner | Certified | No | Yes | | sync_response | Output, Processor | Certified | No | Yes | | a2a_message | Processor | Certified | Yes | Yes | | archiveZIP TAR GZIP | Processor | Certified | No | Yes | | avro | Processor, Scanner | Community | No | Yes | | awk | Processor | Community | No | No | | aws_bedrock_chatAmazon AWS Bedrock Chat | Processor | Certified | No | Yes | | aws_bedrock_embeddingsAmazon AWS Bedrock Embeddings | Processor | Certified | No | Yes | | aws_dynamodb_partiqlAmazon AWS DynamoDB PartiQL | Processor | Certified | No | Yes | | aws_lambdaAWS Lambda Amazon Lambda Lambda | Processor | Certified | No | Yes | | benchmark | Processor | Certified | No | Yes | | bloblang | Processor | Certified | No | Yes | | bounds_check | Processor | Certified | No | Yes | | branch | Processor | Certified | No | Yes | | cached | Processor | Certified | No | Yes | | catch | Processor | Certified | No | Yes | | cohere_chat | Processor | Certified | No | Yes | | cohere_embeddings | Processor | Certified | No | Yes | | cohere_rerank | Processor | Certified | No | Yes | | command | Processor | Certified | No | No | | compress | Processor | Certified | No | Yes | | crash | Processor | Certified | No | No | | decompress | Processor, Scanner | Certified | No | Yes | | dedupe | Processor | Certified | No | Yes | | ffiForeign Function Interface | Processor | Certified | No | No | | for_each | Processor | Certified | No | Yes | | gcp_vertex_ai_chatGCP Vertex AI Google Cloud GCP | Processor | Certified | No | Yes | | gcp_vertex_ai_embeddingsGoogle Cloud GCP | Processor | Certified | No | Yes | | google_drive_download | Processor | Certified | Yes | Yes | | google_drive_list_labels | Processor | Certified | Yes | Yes | | google_drive_search | Processor | Certified | Yes | Yes | | grok | Processor | Community | No | No | | group_by | Processor | Certified | No | Yes | | group_by_value | Processor | Certified | No | Yes | | http | Processor | Certified | No | Yes | | insert_part | Processor | Certified | No | Yes | | javascript | Processor | Certified | No | No | | jmespath | Processor | Certified | No | Yes | | jq | Processor | Certified | No | Yes | | json_schemaJSON Schema | Processor | Certified | No | Yes | | log | Processor | Certified | No | Yes | | mapping | Processor | Certified | No | Yes | | metric | Processor | Certified | No | Yes | | msgpack | Processor | Community | No | No | | mutation | Processor | Certified | No | Yes | | nats_request_replyNATS Request Reply | Processor | Certified | No | Yes | | ollama_chat | Processor | Certified | No | Yes (GPU pipelines only) | | ollama_embeddings | Processor | Certified | No | Yes (GPU pipelines only) | | ollama_moderation | Processor | Certified | No | Yes (GPU pipelines only) | | openai_chat_completion | Processor | Certified | No | Yes | | openai_embeddings | Processor | Certified | No | Yes | | openai_image_generation | Processor | Certified | No | Yes | | openai_speech | Processor | Certified | No | Yes | | openai_transcription | Processor | Certified | No | Yes | | openai_translation | Processor | Certified | No | Yes | | parallel | Processor | Certified | No | Yes | | parquet_decode | Processor | Certified | No | Yes | | parquet_encode | Processor | Certified | No | Yes | | parse_log | Processor | Community | No | Yes | | processors | Processor | Certified | No | Yes | | protobuf | Processor | Certified | No | No | | rate_limit | Processor | Certified | No | Yes | | redis_scriptRedis Script | Processor | Certified | No | Yes | | redpanda_data_transform | Processor | Certified | No | No | | schema_registry_decode | Processor | Certified | No | Yes | | schema_registry_encode | Processor | Certified | No | Yes | | select_parts | Processor | Certified | No | Yes | | sentry_capture | Processor | Community | No | No | | slack_threadSlack Thread | Processor | Certified | Yes | Yes | | sleep | Processor | Certified | No | Yes | | split | Processor | Certified | No | Yes | | string_split | Processor | Certified | No | Yes | | text_chunker | Processor | Certified | No | Yes | | try | Processor | Certified | No | Yes | | try_catch | Processor | Community | No | Yes | | unarchiveZIP TAR GZIP Archive | Processor | Certified | No | Yes | | wasm | Processor | Community | No | No | | while | Processor | Certified | No | Yes | | workflow | Processor | Certified | No | Yes | | xml | Processor | Community | No | Yes | | local | Rate_limit | Certified | No | Yes | | chunker | Scanner | Certified | No | Yes | | json_array | Scanner | Community | No | Yes | | json_documents | Scanner | Certified | No | Yes | | lines | Scanner | Certified | No | Yes | | re_match | Scanner | Certified | No | Yes | | skip_bom | Scanner | Certified | No | Yes | | tar | Scanner | Certified | No | Yes | | to_the_end | Scanner | Certified | No | Yes | | gcp_cloudtraceGCP Cloud Trace | Tracer | Certified | No | Yes | | jaeger | Tracer | Community | No | No | | sql_driver_clickhouseClickHouse | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_gocosmosAzure Cosmos DB | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_driver_mssqlMicrosoft SQL Server | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_driver_mysqlMYSQL | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_oracleOracle | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_postgresPostgreSQL | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_snowflakeSnowflake | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_driver_sqliteSQLite | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_trinoTrino | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | ## [](#about-components)About Components Every Redpanda Connect pipeline has at least one [input](https://docs.redpanda.com/connect/components/inputs/about/), an optional [buffer](https://docs.redpanda.com/connect/components/buffers/about/), an [output](https://docs.redpanda.com/connect/components/outputs/about/) and any number of [processors](https://docs.redpanda.com/connect/components/processors/about/): ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: type: none pipeline: processors: - mapping: | message = this meta.link_count = links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` These are the main components within Redpanda Connect and they provide the majority of useful behavior. ## [](#observability-components)Observability components There are also the observability components: [http](https://docs.redpanda.com/connect/components/http/about/), [logger](https://docs.redpanda.com/connect/components/logger/about/), [metrics](https://docs.redpanda.com/connect/components/metrics/about/), and [tracing](https://docs.redpanda.com/connect/components/tracers/about/), which allow you to specify how Redpanda Connect exposes observability data. ```yaml http: address: 0.0.0.0:4195 enabled: true debug_endpoints: false logger: format: json level: WARN metrics: statsd: address: localhost:8125 flush_period: 100ms tracer: jaeger: agent_address: localhost:6831 ``` ## [](#resource-components)Resource components Finally, there are [caches](https://docs.redpanda.com/connect/components/caches/about/) and [rate limits](https://docs.redpanda.com/connect/components/rate_limits/about/). These are components that are referenced by core components and can be shared. ```yaml input: http_client: # This is an input url: TODO rate_limit: foo_ratelimit # This is a reference to a rate limit pipeline: processors: - cache: # This is a processor resource: baz_cache # This is a reference to a cache operator: add key: '${! json("id") }' value: "x" - mapping: root = if errored() { deleted() } rate_limit_resources: - label: foo_ratelimit local: count: 500 interval: 1s cache_resources: - label: baz_cache memcached: addresses: [ localhost:11211 ] ``` It’s also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the [`resource` input](https://docs.redpanda.com/connect/components/inputs/resource/), [`resource` output](https://docs.redpanda.com/connect/components/outputs/resource/) and [`resource` processor](https://docs.redpanda.com/connect/components/processors/resource/) respectively. For more information about any of these component types check out their sections: - [inputs](https://docs.redpanda.com/connect/components/inputs/about/) - [processors](https://docs.redpanda.com/connect/components/processors/about/) - [outputs](https://docs.redpanda.com/connect/components/outputs/about/) - [buffers](https://docs.redpanda.com/connect/components/buffers/about/) - [metrics](https://docs.redpanda.com/connect/components/metrics/about/) - [tracers](https://docs.redpanda.com/connect/components/tracers/about/) - [logger](https://docs.redpanda.com/connect/components/logger/about/) - [caches](https://docs.redpanda.com/connect/components/caches/about/) - [rate limits](https://docs.redpanda.com/connect/components/rate_limits/about/) ## [](#support-levels)Support levels Each connector has a corresponding level of support, which determines whether it is suitable for production environments or community-driven projects. - **Certified**: Production-ready and available to purchase with Redpanda support. - **Community**: Not considered production-ready and available only for community support in [Slack](https://redpanda.com/slack). Anyone can use the Certified and Community connectors. However, to use Enterprise connectors, you need an [Enterprise Edition license](https://docs.redpanda.com/connect/get-started/licensing/). --- # Page 2: Buffers **URL**: https://docs.redpanda.com/connect/components/buffers/about.md --- # Buffers --- title: Buffers latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: buffers/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/about.adoc description: Learn how buffers can decouple input consumption from processing in Redpanda Connect, and what that means for delivery guarantees. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Redpanda Connect uses a transaction model internally for guaranteeing delivery of messages, this means that a message from an input is not acknowledged (or its offset committed, etc) until that message has been processed and either intentionally deleted or successfully delivered to all outputs. This transaction model makes Redpanda Connect safe to deploy in scenarios where data loss is unacceptable. However, sometimes it’s useful to customize the way in which messages are delivered, and this is where buffers come in. A buffer is an optional component type that comes immediately after the input layer and can be used as a way of decoupling the transaction model from components downstream such as the processing layer and outputs. This is considered an advanced component as most users will likely not benefit from a buffer, but they enable you to do things like group messages using window algorithms or intentionally weaken the delivery guarantees of the pipeline depending on the buffer you choose. Since buffers are able to modify (or disable) the transaction model within Redpanda Connect it is important that when you choose a buffer you read its documentation to understand the implication it will have on delivery guarantees. ## [](#categories)Categories ### Utility Buffers that are intended for niche but general use. [**memory**](https://docs.redpanda.com/connect/components/buffers/memory/) [**sqlite**](https://docs.redpanda.com/connect/components/buffers/sqlite/) ### Windowing Buffers that provide message windowing capabilities. [**system\_window**](https://docs.redpanda.com/connect/components/buffers/system_window/) --- # Page 3: memory **URL**: https://docs.redpanda.com/connect/components/buffers/memory.md --- # memory --- title: memory latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: buffers/memory page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/memory.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/memory.adoc description: Stores consumed messages in memory and acknowledges them at the input level. During shutdown Redpanda Connect will make a best attempt at flushing all remaining messages before exiting cleanly. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores consumed messages in memory and acknowledges them at the input level. During shutdown Redpanda Connect will make a best attempt at flushing all remaining messages before exiting cleanly. This buffer is appropriate when consuming messages from inputs that do not gracefully handle back pressure and where delivery guarantees aren’t critical. This buffer has a configurable limit, where consumption will be stopped with back pressure upstream if the total size of messages in the buffer reaches this amount. Since this calculation is only an estimate, and the real size of messages in RAM is always higher, it is recommended to set the limit significantly below the amount of RAM available. ## [](#delivery-guarantees)Delivery guarantees This buffer intentionally weakens the delivery guarantees of the pipeline and therefore should never be used in places where data loss is unacceptable. ## [](#batching)Batching It is possible to batch up messages sent from this buffer using a [batch policy](https://docs.redpanda.com/connect/configuration/batching/#batch-policy). ### Common ```yml buffer: memory: limit: 524288000 batch_policy: enabled: false count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml buffer: memory: limit: 524288000 batch_policy: enabled: false count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batch_policy)`batch_policy` Optionally configure a policy to flush buffered messages in batches. **Type**: `object` ### [](#batch_policy-byte_size)`batch_policy.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batch_policy-check)`batch_policy.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batch_policy-count)`batch_policy.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batch_policy-enabled)`batch_policy.enabled` Whether to batch messages as they are flushed. **Type**: `bool` **Default**: `false` ### [](#batch_policy-period)`batch_policy.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batch_policy-processors)`batch_policy.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#limit)`limit` The maximum buffer size (in bytes) to allow before applying backpressure upstream. **Type**: `int` **Default**: `524288000` --- # Page 4: none **URL**: https://docs.redpanda.com/connect/components/buffers/none.md --- # none --- title: none latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: buffers/none page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/none.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/none.adoc description: Do not buffer messages. This is the default and most resilient configuration. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Do not buffer messages. This is the default and most resilient configuration. Selecting no buffer means the output layer is directly coupled with the input layer. This is the safest and lowest latency option since acknowledgements from at-least-once protocols can be propagated all the way from the output protocol to the input protocol. If the output layer is hit with back pressure it will propagate all the way to the input layer, and further up the data stream. If you need to relieve your pipeline of this back pressure consider using a more robust buffering solution such as Kafka before resorting to alternatives. ## Common ```yml buffer: none: {} ``` ## Advanced ```yml buffer: none: {} ``` --- # Page 5: sqlite **URL**: https://docs.redpanda.com/connect/components/buffers/sqlite.md --- # sqlite --- title: sqlite latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: buffers/sqlite page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/sqlite.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/sqlite.adoc description: Stores messages in an SQLite database and acknowledges them at the input level. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores messages in an SQLite database and acknowledges them at the input level. Introduced in version 4.11.0. Stored messages are then consumed as a stream from the database and deleted only once they are successfully sent at the output level. If the service is restarted Redpanda Connect will make a best attempt to finish delivering messages that are already read from the database, and when it starts again it will consume from the oldest message that has not yet been delivered. ## [](#delivery-guarantees)Delivery guarantees Messages are not acknowledged at the input level until they have been added to the SQLite database, and they are not removed from the SQLite database until they have been successfully delivered. This means at-least-once delivery guarantees are preserved in cases where the service is shut down unexpectedly. However, since this process relies on interaction with the disk (wherever the SQLite DB is stored) these delivery guarantees are not resilient to disk corruption or loss. ## [](#batching)Batching Messages that are logically batched at the point where they are added to the buffer will continue to be associated with that batch when they are consumed. This buffer is also more efficient when storing messages within batches, and therefore it is recommended to use batching at the input level in high-throughput use cases even if they are not required for processing. ### Common ```yml buffer: sqlite: path: "" # No default (required) pre_processors: [] # No default (optional) post_processors: [] # No default (optional) ``` ### Advanced ```yml buffer: sqlite: path: "" # No default (required) pre_processors: [] # No default (optional) post_processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#path)`path` The path of the database file, which will be created if it does not already exist. **Type**: `string` ### [](#post_processors)`post_processors[]` An optional list of processors to apply to messages after they are consumed from the buffer. These processors are useful for undoing any compression, archiving, etc that may have been done by your `pre_processors`. **Type**: `array` ### [](#pre_processors)`pre_processors[]` An optional list of processors to apply to messages before they are stored within the buffer. These processors are useful for compressing, archiving or otherwise reducing the data in size before it’s stored on disk. **Type**: `array` ## [](#examples)Examples ### [](#batching-for-optimization)Batching for optimization Batching at the input level greatly increases the throughput of this buffer. If logical batches aren’t needed for processing add a [`split` processor](https://docs.redpanda.com/connect/components/processors/split/) to the `post_processors`. ```yaml input: batched: child: sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] policy: count: 100 period: 500ms buffer: sqlite: path: ./foo.db post_processors: - split: {} ``` --- # Page 6: system_window **URL**: https://docs.redpanda.com/connect/components/buffers/system_window.md --- # system_window --- title: system_window latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: buffers/system_window page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/system_window.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/system_window.adoc description: Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is in beta. Its behavior might change in a future release. Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock. Introduced in version 3.53.0. A window is a grouping of messages that fit within a discrete measure of time following the system clock. Messages are allocated to a window either by the processing time (the time at which they’re ingested) or by the event time, and this is controlled via the [`timestamp_mapping` field](#timestamp_mapping). In tumbling mode (default) the beginning of a window immediately follows the end of a prior window. When the buffer is initialized the first window to be created and populated is aligned against the zeroth minute of the zeroth hour of the day by default, and may therefore be open for a shorter period than the specified size. A window is flushed only once the system clock surpasses its scheduled end. If an [`allowed_lateness`](#allowed_lateness) is specified then the window will not be flushed until the scheduled end plus that length of time. When a message is added to a window it has a metadata field `window_end_timestamp` added to it containing the timestamp of the end of the window as an RFC3339 string. ## [](#sliding-windows)Sliding windows Sliding windows begin from an offset of the prior windows' beginning rather than its end, and therefore messages may belong to multiple windows. In order to produce sliding windows specify a [`slide` duration](#slide). ## [](#back-pressure)Back pressure If back pressure is applied to this buffer either due to output services being unavailable or resources being saturated, windows older than the current and last according to the system clock will be dropped in order to prevent unbounded resource usage. This means you should ensure that under the worst case scenario you have enough system memory to store two windows' worth of data at a given time (plus extra for redundancy and other services). If messages could potentially arrive with event timestamps in the future (according to the system clock) then you should also factor in these extra messages in memory usage estimates. ## [](#delivery-guarantees)Delivery guarantees This buffer honours the transaction model within Redpanda Connect in order to ensure that messages are not acknowledged until they are either intentionally dropped or successfully delivered to outputs. However, since messages belonging to an expired window are intentionally dropped there are circumstances where not all messages entering the system will be delivered. When this buffer is configured with a slide duration it is possible for messages to belong to multiple windows, and therefore be delivered multiple times. In this case the first time the message is delivered it will be acked (or nacked) and subsequent deliveries of the same message will be a "best attempt". During graceful termination if the current window is partially populated with messages they will be nacked such that they are re-consumed the next time the service starts. ### Common ```yml buffer: system_window: timestamp_mapping: root = now() size: "" # No default (required) slide: "" offset: "" allowed_lateness: "" ``` ### Advanced ```yml buffer: system_window: timestamp_mapping: root = now() size: "" # No default (required) slide: "" offset: "" allowed_lateness: "" ``` ## [](#examples)Examples ### Counting Passengers at Traffic Given a stream of messages relating to cars passing through various traffic lights of the form: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T09:49:35Z", "registration_plate": "AB1C DEF", "passengers": 3 } ``` We can use a window buffer in order to create periodic messages summarizing the traffic for a period of time of this form: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T10:00:00Z", "total_cars": 15, "passengers": 43 } ``` With the following config: ```yaml buffer: system_window: timestamp_mapping: root = this.created_at size: 1h pipeline: processors: # Group messages of the window into batches of common traffic light IDs - group_by_value: value: '${! json("traffic_light") }' # Reduce each batch to a single message by deleting indexes > 0, and # aggregate the car and passenger counts. - mapping: | root = if batch_index() == 0 { { "traffic_light": this.traffic_light, "created_at": meta("window_end_timestamp"), "total_cars": json("registration_plate").from_all().unique().length(), "passengers": json("passengers").from_all().sum(), } } else { deleted() } ``` ## [](#fields)Fields ### [](#allowed_lateness)`allowed_lateness` An optional duration string describing the length of time to wait after a window has ended before flushing it, allowing late arrivals to be included. Since this windowing buffer uses the system clock an allowed lateness can improve the matching of messages when using event time. **Type**: `string` **Default**: `""` ```yaml # Examples: allowed_lateness: 10s # --- allowed_lateness: 1m ``` ### [](#offset)`offset` An optional duration string to offset the beginning of each window by, otherwise they are aligned to the zeroth minute and zeroth hour on the UTC clock. The offset cannot be a larger or equal measure to the window size or the slide. **Type**: `string` **Default**: `""` ```yaml # Examples: offset: -6h # --- offset: 30m ``` ### [](#size)`size` A duration string describing the size of each window. By default windows are aligned to the zeroth minute and zeroth hour on the UTC clock, meaning windows of 1 hour duration will match the turn of each hour in the day, this can be adjusted with the `offset` field. **Type**: `string` ```yaml # Examples: size: 30s # --- size: 10m ``` ### [](#slide)`slide` An optional duration string describing by how much time the beginning of each window should be offset from the beginning of the previous, and therefore creates sliding windows instead of tumbling. When specified this duration must be smaller than the `size` of the window. **Type**: `string` **Default**: `""` ```yaml # Examples: slide: 30s # --- slide: 10m ``` ### [](#timestamp_mapping)`timestamp_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) applied to each message during ingestion that provides the timestamp to use for allocating it a window. By default the function `now()` is used in order to generate a fresh timestamp at the time of ingestion (the processing time), whereas this mapping can instead extract a timestamp from the message itself (the event time). The timestamp value assigned to `root` must either be a numerical unix time in seconds (with up to nanosecond precision via decimals), or a string in ISO 8601 format. If the mapping fails or provides an invalid result the message will be dropped (with logging to describe the problem). **Type**: `string` **Default**: `root = now()` ```yaml # Examples: timestamp_mapping: root = this.created_at # --- timestamp_mapping: root = meta("kafka_timestamp_unix").number() ``` --- # Page 7: Caches **URL**: https://docs.redpanda.com/connect/components/caches/about.md --- # Caches --- title: Caches latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/about.adoc description: Learn how caches work in Redpanda Connect. A cache is a named key/value store resource that components use for deduplication or data joins. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- A cache is a key/value store which can be used by certain components for applications such as deduplication or data joins. Caches are configured as a named resource: ```yaml cache_resources: - label: foobar memcached: addresses: - localhost:11211 default_ttl: 60s ``` > It’s possible to layer caches with read-through and write-through behavior using the [`multilevel` cache](https://docs.redpanda.com/connect/components/caches/multilevel/). And then any components that use caches have a field `resource` that specifies the cache resource: ```yaml pipeline: processors: - cache: resource: foobar operator: add key: '${! json("message.id") }' value: "storeme" - mapping: root = if errored() { deleted() } ``` For the simple case where you wish to store messages in a cache as an output destination for your pipeline check out the [`cache` output](https://docs.redpanda.com/connect/components/outputs/cache/). To see examples of more advanced uses of caches such as hydration and deduplication check out the [`cache` processor](https://docs.redpanda.com/connect/components/processors/cache/). You can find out more about resources [in this document.](https://docs.redpanda.com/connect/configuration/resources/) --- # Page 8: aws_dynamodb **URL**: https://docs.redpanda.com/connect/components/caches/aws_dynamodb.md --- # aws_dynamodb --- title: aws_dynamodb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/aws_dynamodb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/aws_dynamodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/aws_dynamodb.adoc description: Stores key/value pairs as a single document in a DynamoDB table. The key is stored as a string value and used as the table hash key. The value is stored as a binary value using the data_key field name. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores key/value pairs as a single document in a DynamoDB table. The key is stored as a string value and used as the table hash key. The value is stored as a binary value using the `data_key` field name. Introduced in version 3.36.0. A prefix can be specified to allow multiple cache types to share a single DynamoDB table. An optional TTL duration (`ttl`) and field (`ttl_key`) can be specified if the backing table has TTL enabled. Strong read consistency can be enabled using the `consistent_read` configuration field. #### Common ```yml cache: label: "" aws_dynamodb: table: "" # No default (required) hash_key: "" # No default (required) data_key: "" # No default (required) ``` #### Advanced ```yml cache: label: "" aws_dynamodb: table: "" # No default (required) hash_key: "" # No default (required) data_key: "" # No default (required) consistent_read: false default_ttl: "" # No default (optional) ttl_key: "" # No default (optional) retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#fields)Fields ### [](#consistent_read)`consistent_read` Whether to use strongly consistent reads on Get commands. **Type**: `bool` **Default**: `false` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#data_key)`data_key` The key of the table column to store item values within. **Type**: `string` ### [](#default_ttl)`default_ttl` An optional default TTL to set for items, calculated from the moment the item is cached. A `ttl_key` must be specified in order to set item TTLs. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#hash_key)`hash_key` The key of the table column to store item keys within. **Type**: `string` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#table)`table` The table to store items in. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#ttl_key)`ttl_key` The column key to place the TTL value within. The value is the expiry time as a Unix timestamp in seconds, stored as a number attribute, which is the format that DynamoDB TTL requires. A TTL value is only written when both this field and a TTL duration are set. **Type**: `string` --- # Page 9: aws_s3 **URL**: https://docs.redpanda.com/connect/components/caches/aws_s3.md --- # aws_s3 --- title: aws_s3 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/aws_s3 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/aws_s3.adoc description: Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket. Introduced in version 3.36.0. This cache implements the `add` operation as an existence check followed by a separate upload, so the operation is not atomic and concurrent writers can both succeed. Therefore, this cache is not suitable for deduplication. #### Common ```yml cache: label: "" aws_s3: bucket: "" # No default (required) content_type: application/octet-stream ``` #### Advanced ```yml cache: label: "" aws_s3: bucket: "" # No default (required) content_type: application/octet-stream force_path_style_urls: false retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` It is not possible to atomically upload S3 objects exclusively when the target does not already exist, therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#bucket)`bucket` The S3 bucket to store items in. **Type**: `string` ### [](#content_type)`content_type` The content type to set for each item. **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs, which helps when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 10: couchbase **URL**: https://docs.redpanda.com/connect/components/caches/couchbase.md --- # couchbase --- title: couchbase latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/couchbase page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/couchbase.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/couchbase.adoc description: Use a Couchbase instance as a cache. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Use a Couchbase instance as a cache. Introduced in version 4.12.0. #### Common ```yml cache: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) ``` #### Advanced ```yml cache: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) collection: "" # No default (optional) scope: "" # No default (optional) transcoder: legacy timeout: 15s default_ttl: "" # No default (optional) ``` ## [](#fields)Fields ### [](#bucket)`bucket` The name of the Couchbase bucket to access. **Type**: `string` ### [](#collection)`collection` The name of the collection within the bucket to access. **Type**: `string` ### [](#default_ttl)`default_ttl` An optional expiry applied to documents that are written without a TTL of their own, counted from the moment the document is written. When unset, such documents never expire. **Type**: `string` ### [](#password)`password` A password to authenticate with the Couchbase cluster. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#scope)`scope` Bucket scope. Requires version 4.56.0 or later. **Type**: `string` ### [](#timeout)`timeout` The timeout for Couchbase operations, including connecting to the cluster and waiting for the bucket to become ready. **Type**: `string` **Default**: `15s` ### [](#transcoder)`transcoder` The Couchbase transcoder to use for converting the data format of documents written to and read from Couchbase. **Type**: `string` **Default**: `legacy` | Option | Summary | | --- | --- | | json | JSONTranscoder implements the default transcoding behavior and applies JSON transcoding to all values. This will apply the following behavior to the value: binary ([]byte) → error. default → JSON value, JSON Flags. | | legacy | LegacyTranscoder implements the behavior for a backward-compatible transcoder. This transcoder implements behavior matching that of gocb v1.This will apply the following behavior to the value: binary ([]byte) → binary bytes, Binary expectedFlags. string → string bytes, String expectedFlags. default → JSON value, JSON expectedFlags. | | raw | RawBinaryTranscoder implements passthrough behavior of raw binary data. This transcoder does not apply any serialization. This will apply the following behavior to the value: binary ([]byte) → binary bytes, binary expectedFlags. default → error. | | rawjson | RawJSONTranscoder implements passthrough behavior of JSON data. This transcoder does not apply any serialization. It will forward data across the network without incurring unnecessary parsing costs. This will apply the following behavior to the value: binary ([]byte) → JSON bytes, JSON expectedFlags. string → JSON bytes, JSON expectedFlags. default → error. | | rawstring | RawStringTranscoder implements passthrough behavior of raw string data. This transcoder does not apply any serialization. This will apply the following behavior to the value: string → string bytes, string expectedFlags. default → error. | ### [](#url)`url` The Couchbase connection string. **Type**: `string` ```yaml # Examples: url: couchbase://localhost:11210 ``` ### [](#username)`username` A username to authenticate with the Couchbase cluster. **Type**: `string` --- # Page 11: file **URL**: https://docs.redpanda.com/connect/components/caches/file.md --- # file --- title: file latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/file page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/file.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/file.adoc description: Stores each item in a directory as a file, where an item ID is the path relative to the configured directory. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores each item in a directory as a file, where an item ID is the path relative to the configured directory. This type currently offers no form of item expiry or garbage collection, and is intended to be used for development and debugging purposes only. ```yml cache: label: "" file: directory: "" # No default (required) ``` ## [](#fields)Fields ### [](#directory)`directory` The directory within which to store items. **Type**: `string` --- # Page 12: gcp_cloud_storage **URL**: https://docs.redpanda.com/connect/components/caches/gcp_cloud_storage.md --- # gcp_cloud_storage --- title: gcp_cloud_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/gcp_cloud_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/gcp_cloud_storage.adoc description: Use a Google Cloud Storage bucket as a cache. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Use a Google Cloud Storage bucket as a cache. ```yml cache: label: "" gcp_cloud_storage: bucket: "" # No default (required) content_type: "" # No default (optional) credentials_json: "" ``` It is not possible to atomically upload cloud storage objects exclusively when the target does not already exist, therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#bucket)`bucket` The Google Cloud Storage bucket to store items in. **Type**: `string` ### [](#content_type)`content_type` Optional field to explicitly set the Content-Type. Requires version 4.5.0 or later. **Type**: `string` ### [](#credentials_json)`credentials_json` An optional field to set Google Service Account Credentials json. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` --- # Page 13: lru **URL**: https://docs.redpanda.com/connect/components/caches/lru.md --- # lru --- title: lru latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/lru page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/lru.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/lru.adoc description: Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts. Introduced in version 4.15.0. This provides the lru package which implements a fixed-size thread safe LRU cache. It uses the package [`lru`](https://pkg.go.dev/github.com/hashicorp/golang-lru/v2) The field init\_values can be used to pre-populate the memory cache with any number of key/value pairs: ```yaml cache_resources: - label: foocache lru: cap: 1024 init_values: foo: bar ``` These values can be overridden during execution. #### Common ```yml cache: label: "" lru: cap: 1000 init_values: {} ``` #### Advanced ```yml cache: label: "" lru: cap: 1000 init_values: {} algorithm: standard two_queues_recent_ratio: 0.25 two_queues_ghost_ratio: 0.5 optimistic: false ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` the lru cache implementation **Type**: `string` **Default**: `standard` | Option | Summary | | --- | --- | | arc | is an adaptive replacement cache. It tracks recent evictions as well as recent usage in both the frequent and recent caches. Its computational overhead is comparable to two_queues, but the memory overhead is linear with the size of the cache. ARC has been patented by IBM. | | standard | is a simple LRU cache. It is based on the LRU implementation in groupcache | | two_queues | tracks frequently used and recently used entries separately. This avoids a burst of accesses from taking out frequently used entries, at the cost of about 2x computational overhead and some extra bookkeeping. | ### [](#cap)`cap` The cache maximum capacity (number of entries) **Type**: `int` **Default**: `1000` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `object` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#optimistic)`optimistic` If true, we do not lock on read/write events. The lru package is thread-safe, however the ADD operation is not atomic. **Type**: `bool` **Default**: `false` ### [](#two_queues_ghost_ratio)`two_queues_ghost_ratio` is the default ratio of ghost entries kept to track entries recently evicted on two\_queues cache. **Type**: `float` **Default**: `0.5` ### [](#two_queues_recent_ratio)`two_queues_recent_ratio` is the ratio of the two\_queues cache dedicated to recently added entries that have only been accessed once. **Type**: `float` **Default**: `0.25` --- # Page 14: memcached **URL**: https://docs.redpanda.com/connect/components/caches/memcached.md --- # memcached --- title: memcached latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/memcached page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/memcached.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/memcached.adoc description: Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different namespaces. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different namespaces. #### Common ```yml cache: label: "" memcached: addresses: [] # No default (required) prefix: "" # No default (optional) default_ttl: 300s ``` #### Advanced ```yml cache: label: "" memcached: addresses: [] # No default (required) prefix: "" # No default (optional) default_ttl: 300s retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of addresses of memcached servers to use. **Type**: `array` ### [](#default_ttl)`default_ttl` The expiration applied to items that are written without a TTL of their own. Memcached expirations have a resolution of one second, so any fraction of a second is dropped. **Type**: `string` **Default**: `300s` ### [](#prefix)`prefix` An optional string prepended to every item key, so that multiple caches can share one memcached cluster under different namespaces. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` --- # Page 15: memory **URL**: https://docs.redpanda.com/connect/components/caches/memory.md --- # memory --- title: memory latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/memory page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/memory.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/memory.adoc description: Stores key/value pairs in a map held in memory. This cache is therefore reset every time the service restarts. Each item in the cache has a TTL set from the moment it was last edited, after which it will be removed during the next compaction. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores key/value pairs in a map held in memory. This cache is therefore reset every time the service restarts. Each item in the cache has a TTL set from the moment it was last edited, after which it will be removed during the next compaction. The compaction interval determines how often the cache is cleared of expired items, and this process is only triggered on writes to the cache. Access to the cache is blocked during this process. Item expiry can be disabled entirely by setting the `compaction_interval` to an empty string. The field `init_values` can be used to prepopulate the memory cache with any number of key/value pairs which are exempt from TTLs: ```yaml cache_resources: - label: foocache memory: default_ttl: 60s init_values: foo: bar ``` These values can be overridden during execution, at which point the configured TTL is respected as usual. #### Common ```yml cache: label: "" memory: default_ttl: 5m compaction_interval: 60s init_values: {} ``` #### Advanced ```yml cache: label: "" memory: default_ttl: 5m compaction_interval: 60s init_values: {} shards: 1 ``` ## [](#fields)Fields ### [](#compaction_interval)`compaction_interval` The period of time to wait before each compaction, at which point expired items are removed. This field can be set to an empty string in order to disable compactions/expiry entirely. **Type**: `string` **Default**: `60s` ### [](#default_ttl)`default_ttl` The default TTL of each item. After this period an item will be eligible for removal during the next compaction. **Type**: `string` **Default**: `5m` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `object` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#shards)`shards` A number of logical shards to spread keys across, increasing the shards can have a performance benefit when processing a large number of keys. **Type**: `int` **Default**: `1` --- # Page 16: mongodb **URL**: https://docs.redpanda.com/connect/components/caches/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/mongodb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/mongodb.adoc description: Use a MongoDB instance as a cache. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Use a MongoDB instance as a cache. Introduced in version 3.43.0. #### Common ```yml cache: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) key_field: "" # No default (required) value_field: "" # No default (required) ``` #### Advanced ```yml cache: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos aws: enabled: false region: "" # No default (optional) session_duration: 1h id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) collection: "" # No default (required) key_field: "" # No default (required) value_field: "" # No default (required) ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. Requires version 4.32.0 or later. **Type**: `string` **Default**: `benthos` ### [](#aws)`aws` AWS IAM authentication using the `MONGODB-AWS` mechanism, for example against MongoDB Atlas. When enabled, IAM credentials are used instead of a static username and password. Role-derived session credentials are resolved when the component connects and are re-resolved whenever it reconnects. The `mongodb` processor and cache establish their client once at creation and cannot refresh expiring session credentials, so `role`, `roles` and session tokens are rejected for those components; use the ambient credential chain or long-lived access keys with them. For long-running pipelines, prefer the ambient credential chain (leave keys and roles unset), which the driver refreshes automatically. Requires version 4.106.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication using the driver-native `MONGODB-AWS` mechanism. The MongoDB Atlas database user must be created with the AWS IAM authentication type, and connections require TLS. When no static credentials or roles are configured, the ambient AWS credential chain (environment variables, EC2 instance profile, EKS pod role) is used and expiring credentials are refreshed automatically. **Type**: `bool` **Default**: `false` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region used when assuming roles (for STS calls). Only used when `role` or `roles` are configured; the ambient and static-key paths ignore it. If no region is specified then the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Cannot be combined with `roles`; use the `roles` array instead when chaining multiple roles. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. Cannot be combined with `role`. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-session_duration)`aws.session_duration` The duration of the STS session requested when assuming roles. AWS requires at least 15 minutes and caps sessions created through role chaining at one hour. Only used when `role` or `roles` are configured. For long-running pipelines, prefer the ambient credential chain over a fixed session duration, since the driver refreshes ambient credentials automatically as they near expiry. **Type**: `string` **Default**: `1h` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#collection)`collection` The name of the target collection. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#key_field)`key_field` The field in the document that is used as the key. **Type**: `string` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` ### [](#value_field)`value_field` The field in the document that is used as the value. **Type**: `string` --- # Page 17: multilevel **URL**: https://docs.redpanda.com/connect/components/caches/multilevel.md --- # multilevel --- title: multilevel latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/multilevel page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/multilevel.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/multilevel.adoc description: Combines multiple caches as levels, performing read-through and write-through operations across them. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Combines multiple caches as levels, performing read-through and write-through operations across them. ```yml cache: label: "" multilevel: [] ``` ## [](#examples)Examples ### [](#hot-and-cold-cache)Hot and cold cache The multilevel cache is useful for reducing traffic against a remote cache by routing it through a local cache. In the following example requests will only go through to the memcached server if the local memory cache is missing the key. ```yaml pipeline: processors: - branch: processors: - cache: resource: leveled operator: get key: ${! json("key") } - catch: - mapping: 'root = {"err":error()}' result_map: 'root.result = this' cache_resources: - label: leveled multilevel: [ hot, cold ] - label: hot memory: default_ttl: 60s - label: cold memcached: addresses: [ TODO:11211 ] default_ttl: 60s ``` --- # Page 18: nats_kv **URL**: https://docs.redpanda.com/connect/components/caches/nats_kv.md --- # nats_kv --- title: nats_kv latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/nats_kv page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/nats_kv.adoc description: Cache key/values in a NATS key-value bucket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Cache key/value pairs in a NATS key-value bucket. Introduced in version 4.27.0. #### Common ```yml cache: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) ``` #### Advanced ```yml cache: label: "" nats_kv: urls: [] # No default (required) max_reconnects: 0 # No default (optional) bucket: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 19: noop **URL**: https://docs.redpanda.com/connect/components/caches/noop.md --- # noop --- title: noop latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/noop page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/noop.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/noop.adoc description: Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option. Introduced in version 4.27.0. ```yml cache: label: "" noop: {} ``` --- # Page 20: redis **URL**: https://docs.redpanda.com/connect/components/caches/redis.md --- # redis --- title: redis latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/redis page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/redis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/redis.adoc description: Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration. #### Common ```yml cache: label: "" redis: url: "" # No default (required) prefix: "" # No default (optional) ``` #### Advanced ```yml cache: label: "" redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] prefix: "" # No default (optional) default_ttl: "" # No default (optional) retries: initial_interval: 500ms max_interval: 1s max_elapsed_time: 5s ``` ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#default_ttl)`default_ttl` An optional expiration applied to keys that are written without a TTL of their own. When unset or zero, such keys never expire. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#prefix)`prefix` An optional string added to the start of every key that this cache reads, writes, or deletes, which keeps its keys separate from other data in the same Redis database. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `5s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `1s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 21: redpanda **URL**: https://docs.redpanda.com/connect/components/caches/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/redpanda page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/redpanda.adoc description: A Kafka cache using the Franz Kafka client library. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-10-10" --- A Kafka cache implemented using the [Franz Kafka client library](https://github.com/twmb/franz-go). #### Common ```yaml cache: label: "" redpanda: seed_brokers: [] # No default (required) topic: "" # No default (required) ``` #### Advanced ```yaml cache: label: "" redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) allow_auto_topic_creation: true ``` A cache that stores data in a Kafka topic. This cache is useful for data that is written frequently and queried infrequently. Reads from the cache require scanning the entire topic partition. If you expect frequent access, consider placing an in-memory caching layer in front of this one. Because only the latest values are needed, configure compaction for topics used as caches so that reads are less expensive when topics are rescanned. See [Compaction Settings](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/compaction-settings/). The cache does not have any TTL mechanisms. Use the Kafka topic retention policies to manage TTL. ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. Requires version 4.59.0 or later. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. **Type**: `string` **Default**: `redpanda-connect` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to store data in. **Type**: `string` --- # Page 22: ristretto **URL**: https://docs.redpanda.com/connect/components/caches/ristretto.md --- # ristretto --- title: ristretto latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/ristretto page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/ristretto.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/ristretto.adoc description: Stores key/value pairs in a map held in the memory-bound Ristretto cache. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores key/value pairs in a map held in the memory-bound [Ristretto cache](https://github.com/dgraph-io/ristretto). This cache is more efficient and appropriate for high-volume use cases than the standard memory cache. However, the add command is non-atomic, and therefore this cache is not suitable for deduplication. #### Common ```yml cache: label: "" ristretto: default_ttl: "" ``` #### Advanced ```yml cache: label: "" ristretto: default_ttl: "" get_retries: enabled: false initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#default_ttl)`default_ttl` A default TTL to set for items, calculated from the moment the item is cached. Set to an empty string or zero duration to disable TTLs. **Type**: `string` **Default**: `""` ```yaml # Examples: default_ttl: 5m # --- default_ttl: 60s ``` ### [](#get_retries)`get_retries` Determines how and whether get attempts should be retried if the key is not found. Ristretto is a concurrent cache that does not immediately reflect writes, and so it can sometimes be useful to enable retries at the cost of speed in cases where the key is expected to exist. **Type**: `object` ### [](#get_retries-enabled)`get_retries.enabled` Whether retries should be enabled. **Type**: `bool` **Default**: `false` ### [](#get_retries-initial_interval)`get_retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#get_retries-max_elapsed_time)`get_retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#get_retries-max_interval)`get_retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` --- # Page 23: sql **URL**: https://docs.redpanda.com/connect/components/caches/sql.md --- # sql --- title: sql latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/sql page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/sql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/sql.adoc description: Uses an SQL database table as a destination for storing cache key/value items. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Uses an SQL database table as a destination for storing cache key/value items. Introduced in version 4.26.0. Each cache key/value pair will exist as a row within the specified table. Currently only the key and value columns are set, and therefore any other columns present within the target table must allow NULL values if this cache is going to be used for set and add operations. Cache operations are translated into SQL statements as follows: ## [](#get)Get All `get` operations are performed with a traditional `select` statement. ## [](#delete)Delete All `delete` operations are performed with a traditional `delete` statement. ## [](#set)Set The `set` operation is performed with a traditional `insert` statement. This will behave as an `add` operation by default, and so ideally needs to be adapted in order to provide updates instead of failing on collision s. Since different SQL engines implement upserts differently it is necessary to specify a `set_suffix` that modifies an `insert` statement in order to perform updates on conflict. ## [](#add)Add The `add` operation is performed with a traditional `insert` statement. ### Common ```yml cache: label: "" sql: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) key_column: "" # No default (required) value_column: "" # No default (required) set_suffix: "" # No default (optional) ``` ### Advanced ```yml cache: label: "" sql: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) key_column: "" # No default (required) value_column: "" # No default (required) set_suffix: "" # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) ``` ## [](#fields)Fields ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#key_column)`key_column` The name of a column to be used for storing cache item keys. This column should support strings of arbitrary size. **Type**: `string` ```yaml # Examples: key_column: foo ``` ### [](#set_suffix)`set_suffix` An optional suffix to append to each insert query for a cache `set` operation. This should modify an insert statement into an upsert appropriate for the given SQL engine. **Type**: `string` ```yaml # Examples: set_suffix: ON DUPLICATE KEY UPDATE bar=VALUES(bar) # --- set_suffix: ON CONFLICT (foo) DO UPDATE SET bar=excluded.bar # --- set_suffix: ON CONFLICT (foo) DO NOTHING ``` ### [](#table)`table` The table to insert/read/delete cache items. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#value_column)`value_column` The name of a column to be used for storing cache item values. This column should support strings of arbitrary size. **Type**: `string` ```yaml # Examples: value_column: bar ``` --- # Page 24: ttlru **URL**: https://docs.redpanda.com/connect/components/caches/ttlru.md --- # ttlru --- title: ttlru latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: caches/ttlru page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/ttlru.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/ttlru.adoc description: Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts. Introduced in version 4.15.0. The cache ttlru provides a simple, goroutine safe, cache with a fixed number of entries. Each entry has a per-cache defined TTL. This TTL is reset on both modification and access of the value. As a result, if the cache is full, and no items have expired, when adding a new item, the item with the soonest expiration will be evicted. It uses the package [`expirable`](https://pkg.go.dev/github.com/hashicorp/golang-lru/v2/expirable) The field init\_values can be used to pre-populate the memory cache with any number of key/value pairs: ```yaml cache_resources: - label: foocache ttlru: default_ttl: '5m' cap: 1024 init_values: foo: bar ``` These values can be overridden during execution. #### Common ```yml cache: label: "" ttlru: cap: 1024 default_ttl: 5m0s init_values: {} ``` #### Advanced ```yml cache: label: "" ttlru: cap: 1024 default_ttl: 5m0s ttl: "" # No default (optional) init_values: {} optimistic: false ``` ## [](#fields)Fields ### [](#cap)`cap` The cache maximum capacity (number of entries) **Type**: `int` **Default**: `1024` ### [](#default_ttl)`default_ttl` The cache ttl of each element Requires version 4.21.0 or later. **Type**: `string` **Default**: `5m0s` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `object` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#optimistic)`optimistic` If true, we do not lock on read/write events. The ttlru package is thread-safe, however the ADD operation is not atomic. **Type**: `bool` **Default**: `false` ### [](#ttl)`ttl` Deprecated. Please use `default_ttl` field **Type**: `string` --- # Page 25: HTTP **URL**: https://docs.redpanda.com/connect/components/http/about.md --- # HTTP --- title: HTTP latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: http/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: http/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/http/about.adoc description: Learn about the HTTP server that Redpanda Connect runs for status endpoints and HTTP server components, and how to configure it. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- When Redpanda Connect runs it kicks off an HTTP server that provides a few generally useful endpoints and is also where configured components such as the [`http_server` input](https://docs.redpanda.com/connect/components/inputs/http_server/) [and output](https://docs.redpanda.com/connect/components/outputs/http_server/) can register their own endpoints if they don’t require their own host/port. The configuration for this server lives under the `http` namespace. #### Common ```yml http: enabled: true address: 0.0.0.0:4195 root_path: /benthos debug_endpoints: false ``` #### Advanced ```yml http: enabled: true address: 0.0.0.0:4195 root_path: /benthos debug_endpoints: false cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] basic_auth: enabled: false realm: restricted username: "" password_hash: "" algorithm: sha256 salt: "" ``` The field `enabled` can be set to `false` in order to disable the server. The field `root_path` specifies a general prefix for all endpoints, this can help isolate the service endpoints when using a reverse proxy with other shared services. All endpoints will still be registered at the root as well as behind the prefix, e.g. with a `root_path` set to `/foo` the endpoint `/version` will be accessible from both `/version` and `/foo/version`. ## [](#enabling-https)Enabling HTTPS By default Redpanda Connect will serve traffic over HTTP. In order to enforce TLS and serve traffic exclusively over HTTPS you must provide a `cert_file` and `key_file` path in your config, which point to a file containing a certificate and a matching private key for the server respectively. If the certificate is signed by a certificate authority, the `cert_file` should be the concatenation of the server’s certificate, any intermediates, and the CA’s certificate. ## [](#enabling-basic-authentication)Enabling basic authentication By default Redpanda Connect does not do any sort of authentication for the service-wide HTTP server. However, it’s possible to configure basic authentication with the [`basic_auth`](#basic_auth) field. Passwords configured must be hashed according to the specified algorithm and base64 encoded, for some hashing algorithms you can do this using Redpanda Connect itself: ```sh echo mynewpassword | rpk connect blobl 'root = content().hash("sha256").encode("base64")' ``` ## [](#endpoints)Endpoints The following endpoints will be generally available when the HTTP server is enabled: - `/version` provides version info. - `/ping` can be used as a liveness probe as it always returns a 200. - `/ready` can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned. - `/metrics`, `/stats` both provide metrics when the metrics type is either [`json_api`](https://docs.redpanda.com/connect/components/metrics/json_api/) or [`prometheus`](https://docs.redpanda.com/connect/components/metrics/prometheus/). - `/endpoints` provides a JSON object containing a list of available endpoints, including those registered by configured components. ## [](#cors)CORS In order to serve Cross-Origin Resource Sharing headers, which instruct browsers to allow CORS requests, set the subfield `cors.enabled` to `true`. ### [](#allowed_origins)allowed_origins A list of allowed origins to connect from. The literal value `*` can be specified as a wildcard. Note `cors.enabled` must be set to `true` for this list to take effect. ## [](#debug-endpoints)Debug endpoints The field `debug_endpoints` when set to `true` prompts Redpanda Connect to register a few extra endpoints that can be useful for debugging performance or behavioral problems: - `/debug/config/json` returns the loaded config as JSON. - `/debug/config/yaml` returns the loaded config as YAML. - `/debug/pprof/block` responds with a pprof-formatted block profile. - `/debug/pprof/heap` responds with a pprof-formatted heap profile. - `/debug/pprof/mutex` responds with a pprof-formatted mutex profile. - `/debug/pprof/profile` responds with a pprof-formatted cpu profile. - `/debug/pprof/goroutine` responds with a pprof-formatted goroutine profile. - `/debug/pprof/symbol` looks up the program counters listed in the request, responding with a table mapping program counters to function names. - `/debug/pprof/trace` responds with the execution trace in binary form. Tracing lasts for duration specified in seconds GET parameter, or for 1 second if not specified. - `/debug/stack` returns a snapshot of the current service stack trace. ## [](#fields)Fields ### [](#address)`address` The address to bind to. **Type**: `string` **Default**: `0.0.0.0:4195` ### [](#basic_auth)`basic_auth` Allows you to enforce and customise basic authentication for requests to the HTTP server. **Type**: `object` ### [](#basic_auth-algorithm)`basic_auth.algorithm` Encryption algorithm used to generate `password_hash`. **Type**: `string` **Default**: `sha256` ```yaml # Examples: algorithm: md5 # --- algorithm: sha256 # --- algorithm: bcrypt # --- algorithm: scrypt ``` ### [](#basic_auth-enabled)`basic_auth.enabled` Enable basic authentication **Type**: `bool` **Default**: `false` ### [](#basic_auth-password_hash)`basic_auth.password_hash` Hashed password required to authenticate. (base64 encoded) **Type**: `string` **Default**: `""` ### [](#basic_auth-realm)`basic_auth.realm` Custom realm name **Type**: `string` **Default**: `restricted` ### [](#basic_auth-salt)`basic_auth.salt` Salt for scrypt algorithm. (base64 encoded) **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` Username required to authenticate. **Type**: `string` **Default**: `""` ### [](#cert_file)`cert_file` An optional certificate file for enabling TLS. **Type**: `string` **Default**: `""` ### [](#cors-2)`cors` Adds Cross-Origin Resource Sharing headers. Requires version 3.63.0 or later. **Type**: `object` ### [](#cors-allowed_origins)`cors.allowed_origins[]` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#debug_endpoints)`debug_endpoints` Whether to register a few extra endpoints that can be useful for debugging performance or behavioral problems. **Type**: `bool` **Default**: `false` ### [](#enabled)`enabled` Whether to enable to HTTP server. **Type**: `bool` **Default**: `true` ### [](#key_file)`key_file` An optional key file for enabling TLS. **Type**: `string` **Default**: `""` ### [](#root_path)`root_path` Specifies a general prefix for all endpoints, this can help isolate the service endpoints when using a reverse proxy with other shared services. All endpoints will still be registered at the root as well as behind the prefix, e.g. with a root\_path set to `/foo` the endpoint `/version` will be accessible from both `/version` and `/foo/version`. **Type**: `string` **Default**: `/benthos` --- # Page 26: Inputs **URL**: https://docs.redpanda.com/connect/components/inputs/about.md --- # Inputs --- title: Inputs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/about.adoc description: Learn how inputs work in Redpanda Connect. An input is a source of data piped through an optional array of processors. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- An input is a source of data piped through an array of optional [processors](https://docs.redpanda.com/connect/components/processors/about/): ```yaml input: label: my_redis_input redis_streams: url: tcp://localhost:6379 streams: - benthos_stream body_key: body consumer_group: benthos_group # Optional list of processing steps processors: - mapping: | root.document = this.without("links") root.link_count = this.links.length() ``` Some inputs have a logical end, for example a [`csv` input](https://docs.redpanda.com/connect/components/inputs/csv/) ends once the last row is consumed, when this happens the input gracefully terminates and Redpanda Connect will shut itself down once all messages have been processed fully. It’s also possible to specify a logical end for an input that otherwise doesn’t have one with the [`read_until` input](https://docs.redpanda.com/connect/components/inputs/read_until/), which checks a condition against each consumed message in order to determine whether it should be the last. ## [](#brokering)Brokering Only one input is configured at the root of a Redpanda Connect config. However, the root input can be a [broker](https://docs.redpanda.com/connect/components/inputs/broker/) which combines multiple inputs and merges the streams: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup - redis_streams: url: tcp://localhost:6379 streams: - benthos_stream body_key: body consumer_group: benthos_group ``` ## [](#labels)Labels Inputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple inputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](https://docs.redpanda.com/connect/components/metrics/about/). ### [](#sequential-reads)Sequential reads Sometimes it’s useful to consume a sequence of inputs, where an input is only consumed once its predecessor is drained fully, you can achieve this with the [`sequence` input](https://docs.redpanda.com/connect/components/inputs/sequence/). ## [](#generating-messages)Generating messages It’s possible to generate data with Redpanda Connect using the [`generate` input](https://docs.redpanda.com/connect/components/inputs/generate/), which is also a convenient way to trigger scheduled pipelines. ## [](#categories)Categories ### Services Inputs that consume from storage or message streaming services. [**RabbitMQ**](https://docs.redpanda.com/connect/components/inputs/amqp_0_9/) [**RabbitMQ**](https://docs.redpanda.com/connect/components/inputs/amqp_1/) [**AWS CloudWatch Logs**](https://docs.redpanda.com/connect/components/inputs/aws_cloudwatch_logs/) [**Amazon DynamoDB CDC**](https://docs.redpanda.com/connect/components/inputs/aws_dynamodb_cdc/) [**AWS Kinesis**](https://docs.redpanda.com/connect/components/inputs/aws_kinesis/) [**AWS S3**](https://docs.redpanda.com/connect/components/inputs/aws_s3/) [**AWS SQS**](https://docs.redpanda.com/connect/components/inputs/aws_sqs/) [**Azure Blob Storage**](https://docs.redpanda.com/connect/components/inputs/azure_blob_storage/) [**Azure Queue Storage**](https://docs.redpanda.com/connect/components/inputs/azure_queue_storage/) [**Azure Table Storage**](https://docs.redpanda.com/connect/components/inputs/azure_table_storage/) [**beanstalkd**](https://docs.redpanda.com/connect/components/inputs/beanstalkd/) [**Apache Cassandra**](https://docs.redpanda.com/connect/components/inputs/cassandra/) [**cockroachdb\_changefeed**](https://docs.redpanda.com/connect/components/inputs/cockroachdb_changefeed/) [**discord**](https://docs.redpanda.com/connect/components/inputs/discord/) [**GCP BigQuery**](https://docs.redpanda.com/connect/components/inputs/gcp_bigquery_select/) [**GCP Cloud Storage**](https://docs.redpanda.com/connect/components/inputs/gcp_cloud_storage/) [**GCP PubSub**](https://docs.redpanda.com/connect/components/inputs/gcp_pubsub/) [**Google Cloud**](https://docs.redpanda.com/connect/components/inputs/gcp_spanner_cdc/) [**git**](https://docs.redpanda.com/connect/components/inputs/git/) [**hdfs**](https://docs.redpanda.com/connect/components/inputs/hdfs/) [**Jira**](https://docs.redpanda.com/connect/components/inputs/jira/) [**microsoft\_sql\_server\_cdc**](https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc/) [**MongoDB**](https://docs.redpanda.com/connect/components/inputs/mongodb/) [**MongoDB CDC**](https://docs.redpanda.com/connect/components/inputs/mongodb_cdc/) [**mqtt**](https://docs.redpanda.com/connect/components/inputs/mqtt/) [**mysql\_cdc**](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/) [**NATS**](https://docs.redpanda.com/connect/components/inputs/nats/) [**NATS JetStream**](https://docs.redpanda.com/connect/components/inputs/nats_jetstream/) [**NATS KV**](https://docs.redpanda.com/connect/components/inputs/nats_kv/) [**NATS Stream**](https://docs.redpanda.com/connect/components/inputs/nats_stream/) [**nsq**](https://docs.redpanda.com/connect/components/inputs/nsq/) [**ockam\_kafka**](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/) [**Oracle CDC**](https://docs.redpanda.com/connect/components/inputs/oracledb_cdc/) [**OpenTelemetry**](https://docs.redpanda.com/connect/components/inputs/otlp_grpc/) [**OpenTelemetry**](https://docs.redpanda.com/connect/components/inputs/otlp_http/) [**postgres\_cdc**](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) [**Apache Pulsar**](https://docs.redpanda.com/connect/components/inputs/pulsar/) [**Redis List**](https://docs.redpanda.com/connect/components/inputs/redis_list/) [**Redis PubSub**](https://docs.redpanda.com/connect/components/inputs/redis_pubsub/) [**Redis**](https://docs.redpanda.com/connect/components/inputs/redis_scan/) [**Redis Streams**](https://docs.redpanda.com/connect/components/inputs/redis_streams/) [**redpanda**](https://docs.redpanda.com/connect/components/inputs/redpanda/) [**redpanda\_migrator**](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) [**Salesforce**](https://docs.redpanda.com/connect/components/inputs/salesforce/) [**Salesforce**](https://docs.redpanda.com/connect/components/inputs/salesforce_cdc/) [**Salesforce**](https://docs.redpanda.com/connect/components/inputs/salesforce_graphql/) [**SAP HANA**](https://docs.redpanda.com/connect/components/inputs/sap_hana/) [**spicedb\_watch**](https://docs.redpanda.com/connect/components/inputs/spicedb_watch/) [**splunk**](https://docs.redpanda.com/connect/components/inputs/splunk/) [**SQL**](https://docs.redpanda.com/connect/components/inputs/sql_raw/) [**SQL**](https://docs.redpanda.com/connect/components/inputs/sql_select/) [**tigerbeetle\_cdc**](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/) [**timeplus**](https://docs.redpanda.com/connect/components/inputs/timeplus/) [**twitter\_search**](https://docs.redpanda.com/connect/components/inputs/twitter_search/) ### AWS Inputs that consume from Amazon Web Services products. [**AWS CloudWatch Logs**](https://docs.redpanda.com/connect/components/inputs/aws_cloudwatch_logs/) [**Amazon DynamoDB CDC**](https://docs.redpanda.com/connect/components/inputs/aws_dynamodb_cdc/) [**AWS Kinesis**](https://docs.redpanda.com/connect/components/inputs/aws_kinesis/) [**AWS S3**](https://docs.redpanda.com/connect/components/inputs/aws_s3/) [**AWS SQS**](https://docs.redpanda.com/connect/components/inputs/aws_sqs/) ### Azure Inputs that consume from Microsoft Azure services. [**Azure Blob Storage**](https://docs.redpanda.com/connect/components/inputs/azure_blob_storage/) [**Microsoft Azure**](https://docs.redpanda.com/connect/components/inputs/azure_cosmosdb/) [**Azure Queue Storage**](https://docs.redpanda.com/connect/components/inputs/azure_queue_storage/) [**Azure Table Storage**](https://docs.redpanda.com/connect/components/inputs/azure_table_storage/) ### Utility Inputs that provide utility by generating data or combining/wrapping other inputs. [**batched**](https://docs.redpanda.com/connect/components/inputs/batched/) [**broker**](https://docs.redpanda.com/connect/components/inputs/broker/) [**dynamic**](https://docs.redpanda.com/connect/components/inputs/dynamic/) [**generate**](https://docs.redpanda.com/connect/components/inputs/generate/) [**inproc**](https://docs.redpanda.com/connect/components/inputs/inproc/) [**read\_until**](https://docs.redpanda.com/connect/components/inputs/read_until/) [**resource**](https://docs.redpanda.com/connect/components/inputs/resource/) [**sequence**](https://docs.redpanda.com/connect/components/inputs/sequence/) [**subprocess**](https://docs.redpanda.com/connect/components/inputs/subprocess/) ### Local Inputs that consume from the local machine/filesystem. [**CSV**](https://docs.redpanda.com/connect/components/inputs/csv/) [**file**](https://docs.redpanda.com/connect/components/inputs/file/) [**parquet**](https://docs.redpanda.com/connect/components/inputs/parquet/) [**stdin**](https://docs.redpanda.com/connect/components/inputs/stdin/) ### Social Inputs that consume from social applications and services. [**discord**](https://docs.redpanda.com/connect/components/inputs/discord/) [**twitter\_search**](https://docs.redpanda.com/connect/components/inputs/twitter_search/) ### GCP Inputs that consume from Google Cloud Platform services. [**GCP BigQuery**](https://docs.redpanda.com/connect/components/inputs/gcp_bigquery_select/) [**GCP Cloud Storage**](https://docs.redpanda.com/connect/components/inputs/gcp_cloud_storage/) [**GCP PubSub**](https://docs.redpanda.com/connect/components/inputs/gcp_pubsub/) [**Google Cloud**](https://docs.redpanda.com/connect/components/inputs/gcp_spanner_cdc/) ### Network Inputs that consume directly from low level network protocols. [**HTTP**](https://docs.redpanda.com/connect/components/inputs/http_client/) [**HTTP**](https://docs.redpanda.com/connect/components/inputs/http_server/) [**nanomsg**](https://docs.redpanda.com/connect/components/inputs/nanomsg/) [**OpenTelemetry**](https://docs.redpanda.com/connect/components/inputs/otlp_grpc/) [**OpenTelemetry**](https://docs.redpanda.com/connect/components/inputs/otlp_http/) [**sftp**](https://docs.redpanda.com/connect/components/inputs/sftp/) [**socket**](https://docs.redpanda.com/connect/components/inputs/socket/) [**socket\_server**](https://docs.redpanda.com/connect/components/inputs/socket_server/) [**websocket**](https://docs.redpanda.com/connect/components/inputs/websocket/) [**zmq4**](https://docs.redpanda.com/connect/components/inputs/zmq4/) ### Integration [**schema\_registry**](https://docs.redpanda.com/connect/components/inputs/schema_registry/) ### SpiceDB [**spicedb\_watch**](https://docs.redpanda.com/connect/components/inputs/spicedb_watch/) --- # Page 27: amqp_0_9 **URL**: https://docs.redpanda.com/connect/components/inputs/amqp_0_9.md --- # amqp_0_9 --- title: amqp_0_9 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/amqp_0_9 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/amqp_0_9.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/amqp_0_9.adoc description: Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. #### Common ```yml input: label: "" amqp_0_9: urls: [] # No default (required) queue: "" # No default (required) consumer_tag: "" prefetch_count: 10 ``` #### Advanced ```yml input: label: "" amqp_0_9: urls: [] # No default (required) queue: "" # No default (required) queue_declare: enabled: false durable: true auto_delete: false arguments: {} # No default (optional) bindings_declare: [] # No default (optional) consumer_tag: "" auto_ack: false nack_reject_patterns: [] prefetch_count: 10 prefetch_size: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` TLS is automatically enabled when connecting to an `amqps` URL. However, you can customize [TLS settings](#tls) if required. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `amqp_content_type` - `amqp_content_encoding` - `amqp_delivery_mode` - `amqp_priority` - `amqp_correlation_id` - `amqp_reply_to` - `amqp_expiration` - `amqp_message_id` - `amqp_timestamp` - `amqp_type` - `amqp_user_id` - `amqp_app_id` - `amqp_consumer_tag` - `amqp_delivery_tag` - `amqp_redelivered` - `amqp_exchange` - `amqp_routing_key` - All existing message headers, including nested headers prefixed with the key of their respective parent. You can access these metadata fields using [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_ack)`auto_ack` Set to `true` to automatically acknowledge messages as soon as they are consumed rather than waiting for acknowledgments from downstream. This can improve throughput and prevent the pipeline from becoming blocked, but delivery guarantees are lost. **Type**: `bool` **Default**: `false` ### [](#bindings_declare)`bindings_declare[]` Declares the bindings of the target queue to make sure they exist and are configured correctly. **Type**: `array` ```yaml # Examples: bindings_declare: - exchange: foo key: bar ``` ### [](#bindings_declare-exchange)`bindings_declare[].exchange` The exchange of the declared binding. **Type**: `string` **Default**: `""` ### [](#bindings_declare-key)`bindings_declare[].key` The key of the declared binding. **Type**: `string` **Default**: `""` ### [](#consumer_tag)`consumer_tag` A consumer tag to uniquely identify the consumer. **Type**: `string` **Default**: `""` ### [](#nack_reject_patterns)`nack_reject_patterns[]` A list of regular expression patterns to match against errors in messages that Redpanda Connect fails to deliver. When a message has an error that matches a pattern, it is dropped or delivered to a dead-letter queue (if a queue has been configured). By default, failed messages are negatively acknowledged (nacked) and requeued. Requires version 3.64.0 or later. **Type**: `array` **Default**: `[]` ```yaml # Examples: nack_reject_patterns: - "^reject me please:.+$" ``` ### [](#prefetch_count)`prefetch_count` The maximum number of pending messages at a given time. **Type**: `int` **Default**: `10` ### [](#prefetch_size)`prefetch_size` The maximum size (in bytes) of pending messages to have consumed at a time. **Type**: `int` **Default**: `0` ### [](#queue)`queue` An AMQP queue to consume from. **Type**: `string` ### [](#queue_declare)`queue_declare` Declares the target queue (`queue`) to make sure a queue with the specified name exists and is configured correctly. If the queue does not exist, it is created. If the queue already exists, the declaration verifies that the fields specified in this object match its properties. **Type**: `object` ### [](#queue_declare-arguments)`queue_declare.arguments` Arguments for server-specific implementations of the queue (optional). You can use arguments to configure additional parameters for queue types that require them. For more information about available arguments, see the [RabbitMQ Client Library](https://github.com/rabbitmq/amqp091-go/blob/b3d409fe92c34bea04d8123a136384c85e8dc431/types.go#L282-L362). > 📝 **NOTE** > > This input sends every argument value as a string. RabbitMQ requires `x-max-length`, `x-max-length-bytes`, `x-message-ttl`, `x-expires`, `x-consumer-timeout`, and `x-stream-max-segment-size-bytes` to be integers, and `x-single-active-consumer` to be a boolean, so declaring a queue with any of them fails. Only string arguments, such as `x-queue-type`, `x-overflow`, and `x-max-age`, work. | Argument | Description | Accepted values | | --- | --- | --- | | x-queue-type | Declares the type of queue. | Options: classic (default), quorum, and stream. | | x-max-length | The maximum number of messages in the queue. | A non-negative integer. | | x-max-length-bytes | The maximum size of messages (in bytes) in the queue. | A non-negative integer. | | x-overflow | Sets the queue’s overflow behavior. | Options: drop-head (default), reject-publish, reject-publish-dlx. | | x-message-ttl | The duration (in milliseconds) that messages remain in the queue before they expire and are discarded. | A string that represents the number of milliseconds. For example, 60000 retains messages for one minute. | | x-expires | The duration (in milliseconds) after which the queue automatically expires. | A positive integer that represents the number of milliseconds. | | x-max-age | The duration (in configurable units) that streamed messages are retained on disk before they are discarded. | Options: Y, M, D, h, m, s. For example, 7D retains messages for a week. | | x-stream-max-segment-size-bytes | The maximum size (in bytes) of the segment files held on disk. | A positive integer. Default: 500000000 (approximately 500 MB). | | x-queue-version | The version of the classic queue to use. | Options: 2. RabbitMQ 4.3.0 and later accept only 2, and earlier versions also accept 1. | | x-consumer-timeout | The duration (in milliseconds) that a consumer can remain idle before it is automatically canceled. | A positive integer that represents the number of milliseconds. For example, 60000 sets a timeout duration of one minute. | | x-single-active-consumer | When set to true, a single consumer receives messages from the queue even when multiple consumers are subscribed to it. | A boolean. | Requires version 4.46.0 or later. **Type**: `object` ```yaml # Examples: arguments: x-overflow: reject-publish x-queue-type: quorum ``` ### [](#queue_declare-auto_delete)`queue_declare.auto_delete` Whether the declared queue auto-deletes when there are no active consumers. Requires version 4.5.0 or later. **Type**: `bool` **Default**: `false` ### [](#queue_declare-durable)`queue_declare.durable` Whether the declared queue is durable. **Type**: `bool` **Default**: `true` ### [](#queue_declare-enabled)`queue_declare.enabled` Whether to enable queue declaration. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of AMQP 0.9.1 broker URLs to connect to, in the form `amqp://user:password@host:port/vhost`. Each URL is tried in order until a connection is established, and that URL is then used until the connection is closed. When `tls` is enabled and a URL has no user credentials, the connection authenticates with the SASL `EXTERNAL` mechanism using the client certificate. If an item in the list contains commas, it is split into multiple URLs. Requires version 3.58.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 28: amqp_1 **URL**: https://docs.redpanda.com/connect/components/inputs/amqp_1.md --- # amqp_1 --- title: amqp_1 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/amqp_1 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/amqp_1.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/amqp_1.adoc description: Reads messages from an AMQP (1.0) server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads messages from an AMQP (1.0) server. #### Common ```yml input: label: "" amqp_1: urls: [] # No default (optional) source_address: "" # No default (required) ``` #### Advanced ```yml input: label: "" amqp_1: urls: [] # No default (optional) source_address: "" # No default (required) azure_renew_lock: false read_header: false credit: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `amqp_content_type` - `amqp_content_encoding` - `amqp_creation_time` - All string typed message annotations You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). By setting `read_header` to `true`, additional message header properties will be added to each message: - `amqp_durable` - `amqp_priority` - `amqp_ttl` - `amqp_first_acquirer` - `amqp_delivery_count` ## [](#performance)Performance This input benefits from receiving multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages with the field `credit`. ## [](#fields)Fields ### [](#azure_renew_lock)`azure_renew_lock` Experimental: Azure service bus specific option to renew lock if processing takes more then configured lock time Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#credit)`credit` Specifies the maximum number of unacknowledged messages the sender can transmit. Once this limit is reached, no more messages will arrive until messages are acknowledged and settled. Requires version 4.26.0 or later. **Type**: `int` **Default**: `64` ### [](#read_header)`read_header` Read additional message header fields into `amqp_*` metadata properties. Requires version 4.25.0 or later. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism to use. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | anonymous | Anonymous SASL authentication. | | none | No SASL based authentication. | | plain | Plain text SASL authentication. | ### [](#sasl-password)`sasl.password` A SASL plain text password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-user)`sasl.user` A SASL plain text username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#source_address)`source_address` The source address to consume from. **Type**: `string` ```yaml # Examples: source_address: /foo # --- source_address: queue:/bar # --- source_address: topic:/baz ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of AMQP 1.0 broker URLs to connect to. Each URL is tried in order until a connection is established, and that URL is then used until the connection is closed. This field replaces the deprecated `url` field, which is only read when `urls` is empty. If an item in the list contains commas, it is split into multiple URLs. Requires version 4.23.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 29: aws_cloudwatch_logs **URL**: https://docs.redpanda.com/connect/components/inputs/aws_cloudwatch_logs.md --- # aws_cloudwatch_logs --- title: aws_cloudwatch_logs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/aws_cloudwatch_logs page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_cloudwatch_logs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_cloudwatch_logs.adoc description: Consumes log events from AWS CloudWatch Logs. page-git-created-date: "2026-03-13" page-git-modified-date: "2026-10-10" --- Consumes log events from AWS CloudWatch Logs. Introduced in version 4.81.0. #### Common ```yml input: label: "" aws_cloudwatch_logs: log_group_name: "" # No default (required) log_stream_names: [] # No default (optional) log_stream_prefix: "" # No default (optional) filter_pattern: "" # No default (optional) start_time: "" # No default (optional) poll_interval: 5s auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" aws_cloudwatch_logs: log_group_name: "" # No default (required) log_stream_names: [] # No default (optional) log_stream_prefix: "" # No default (optional) filter_pattern: "" # No default (optional) start_time: "" # No default (optional) poll_interval: 5s limit: 1000 structured_log: true api_timeout: 30s auto_replay_nacks: true region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Polls CloudWatch Log Groups for log events. Supports filtering by log streams, CloudWatch filter patterns, and configurable start times. Each log event becomes a separate message with metadata including the log group name, log stream name, timestamp, and ingestion time. > ❗ **IMPORTANT** > > This input provides at-least-once delivery. It tracks its position in memory only, so if the process restarts, it resumes from the configured `start_time` (or the beginning if not set). Duplicates can occur across restarts. For exactly-once outcomes, implement idempotent or deduplicated downstream processing. ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `cloudwatch_log_group` - The name of the log group - `cloudwatch_log_stream` - The name of the log stream - `cloudwatch_timestamp` - The timestamp of the log event (Unix milliseconds) - `cloudwatch_ingestion_time` - The ingestion timestamp (Unix milliseconds) - `cloudwatch_event_id` - The unique event ID You can access these metadata fields using [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/). ## [](#fields)Fields ### [](#api_timeout)`api_timeout` The maximum time to wait for an API request to complete. **Type**: `string` **Default**: `30s` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#filter_pattern)`filter_pattern` An optional CloudWatch Logs filter pattern to apply when querying log events. For syntax details, see the [CloudWatch Logs filter and pattern syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html) documentation. **Type**: `string` ```yaml # Examples: filter_pattern: "[ERROR]" ``` ### [](#limit)`limit` The maximum number of log events to return in a single API call. Valid range: 1-10000. **Type**: `int` **Default**: `1000` ### [](#log_group_name)`log_group_name` The name of the CloudWatch Log Group to consume from. **Type**: `string` ```yaml # Examples: log_group_name: my-app-logs ``` ### [](#log_stream_names)`log_stream_names[]` An optional list of log stream names to consume from. If not set, events from all streams in the log group will be consumed. **Type**: `array` ```yaml # Examples: log_stream_names: - stream-1 - stream-2 ``` ### [](#log_stream_prefix)`log_stream_prefix` An optional log stream name prefix to filter streams. Only streams starting with this prefix will be consumed. **Type**: `string` ```yaml # Examples: log_stream_prefix: prod- ``` ### [](#poll_interval)`poll_interval` The interval at which to poll for new log events. **Type**: `string` **Default**: `5s` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#start_time)`start_time` The time to start consuming log events from. Can be an RFC3339 timestamp (for example, `2024-01-01T00:00:00Z`) or the string `now` to start consuming from the current time. If not set, starts from the beginning of available logs. **Type**: `string` ```yaml # Examples: start_time: 2024-01-01T00:00:00Z # --- start_time: now ``` ### [](#structured_log)`structured_log` Whether to output log events as structured JSON objects with all metadata fields, or as plain text messages with metadata stored in Redpanda Connect message metadata. **Type**: `bool` **Default**: `true` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 30: aws_dynamodb_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/aws_dynamodb_cdc.md --- # aws_dynamodb_cdc --- title: aws_dynamodb_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/aws_dynamodb_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_dynamodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_dynamodb_cdc.adoc description: Reads change data capture (CDC) events from DynamoDB Streams. page-topic-type: reference personas: data_engineer, streaming_developer, platform_operator learning-objective-1: Look up configuration options for DynamoDB CDC streaming learning-objective-2: Find metadata fields available for message processing learning-objective-3: Identify checkpointing and performance tuning settings page-git-created-date: "2026-03-02" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Stream item-level changes from DynamoDB tables using DynamoDB Streams. This input automatically manages shards, checkpoints progress for recovery, and processes multiple shards concurrently. Introduced in version 4.79.0. Use this reference to: - Look up configuration options for DynamoDB CDC streaming - Find metadata fields available for message processing - Identify checkpointing and performance tuning settings ### Common ```yml input: label: "" aws_dynamodb_cdc: tables: [] checkpoint_table: redpanda_dynamodb_checkpoints checkpoint_namespace: "" start_from: trim_horizon auto_replay_nacks: true snapshot_mode: none ``` ### Advanced ```yml input: label: "" aws_dynamodb_cdc: tables: [] table_discovery_mode: single table_tag_filter: "" table_discovery_interval: 5m checkpoint_table: redpanda_dynamodb_checkpoints checkpoint_namespace: "" global_table: false global_table_replicas: [] batch_size: 1000 poll_interval: 1s start_from: trim_horizon checkpoint_limit: 1000 max_tracked_shards: 10000 auto_replay_nacks: true throttle_backoff: 100ms snapshot_mode: none snapshot_segments: 1 snapshot_batch_size: 100 snapshot_throttle: 100ms snapshot_deduplicate: true snapshot_buffer_size: 100000 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#prerequisites)Prerequisites The source DynamoDB table must have [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) enabled. You can enable streams with one of these view types: - `KEYS_ONLY`: Only the key attributes of the modified item - `NEW_IMAGE`: The entire item as it appears after the modification - `OLD_IMAGE`: The entire item as it appeared before the modification - `NEW_AND_OLD_IMAGES`: Both the new and old item images ## [](#checkpointing)Checkpointing Checkpoints are stored in a separate DynamoDB table (configured via `checkpoint_table`). This table is created automatically if it does not exist. On restart, the input resumes from the last checkpointed position for each shard. ### [](#share-a-checkpoint-table-across-pipelines)Share a checkpoint table across pipelines By default, checkpoints are keyed by stream and shard only, so multiple pipelines that read the same stream and share one `checkpoint_table` overwrite each other’s positions, which causes skipped or duplicated events. This is common when several developers or environments test against the same table. Set `checkpoint_namespace` to give each pipeline its own isolated set of checkpoints within a shared table. Redpanda Connect prefixes the namespace to the checkpoint key, so pipelines with different namespaces never collide: ```yaml # Alice's pipeline input: aws_dynamodb_cdc: tables: [ orders ] region: us-east-1 checkpoint_table: shared_dynamodb_checkpoints checkpoint_namespace: dev-alice start_from: trim_horizon ``` ```yaml # Bob's pipeline: same stream and checkpoint table, isolated by namespace input: aws_dynamodb_cdc: tables: [ orders ] region: us-east-1 checkpoint_table: shared_dynamodb_checkpoints checkpoint_namespace: dev-bob start_from: trim_horizon ``` The `checkpoint_namespace` field requires Redpanda Connect 4.101.0 or later. It is backward compatible: leaving it unset (the default) keeps the existing checkpoint keys unchanged, so existing deployments are unaffected. The value cannot contain a `#` character. > 📝 **NOTE** > > A namespace provides isolation, not coordination. Two pipelines that use the same `checkpoint_namespace` and `checkpoint_table` still overwrite each other’s checkpoints. To run readers independently, give each one a distinct namespace. > ⚠️ **WARNING** > > Changing or removing `checkpoint_namespace` makes the pipeline read checkpoints under the new key. If no checkpoints exist yet for that namespace, for example the first time you set it or when you switch to a namespace that has never been used, the pipeline has nothing to resume from and starts at `start_from` (with `start_from: trim_horizon`, this replays all records still available in the stream’s 24-hour retention window). Switching back to a previously used namespace resumes from that namespace’s last checkpoints. ## [](#alternative-components)Alternative components For better performance and longer retention (up to 1 year vs 24 hours), consider using Kinesis Data Streams for DynamoDB with the `aws_kinesis` input instead. ## [](#message-structure)Message structure Each CDC event is delivered as a JSON message with the following structure. Use these fields in your Bloblang mappings with `this.`: ```json { "eventID": "abc123-", (1) "eventName": "INSERT | MODIFY | REMOVE", (2) "eventSource": "aws:dynamodb", "awsRegion": "us-east-1", "tableName": "my-table", (3) "dynamodb": { "keys": { (4) "pk": "user#123", "sk": "profile" }, "newImage": { (5) "pk": "user#123", "sk": "profile", "name": "Alice", "email": "alice@example.com" }, "oldImage": { (6) "pk": "user#123", "sk": "profile", "name": "Alice Smith" }, "sequenceNumber": "12345678901234567890", (7) "sizeBytes": 256, "streamViewType": "NEW_AND_OLD_IMAGES" } } ``` | 1 | Unique identifier for this change event. | | --- | --- | | 2 | Type of change: INSERT (new item), MODIFY (updated item), or REMOVE (deleted item). | | 3 | Name of the source DynamoDB table. | | 4 | Primary key attributes of the changed item. Always present. | | 5 | Item state after the change. Present for INSERT and MODIFY events (requires NEW_IMAGE or NEW_AND_OLD_IMAGES stream view type). | | 6 | Item state before the change. Present for MODIFY and REMOVE events (requires OLD_IMAGE or NEW_AND_OLD_IMAGES stream view type). | | 7 | Position of this record in the shard, used for ordering and checkpointing. | > 📝 **NOTE** > > DynamoDB attribute values are automatically unmarshalled from DynamoDB’s type format (`{"S": "value"}`) to plain values (`"value"`). ### [](#example-mapping)Example mapping ```yaml pipeline: processors: - mapping: | root.event_type = this.eventName root.table = this.tableName root.keys = this.dynamodb.keys root.new_data = this.dynamodb.newImage root.old_data = this.dynamodb.oldImage ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `dynamodb_shard_id` - The shard ID from which the record was read (empty for snapshot records) - `dynamodb_sequence_number` - The sequence number of the record in the stream (empty for snapshot records) - `dynamodb_approximate_creation_time` - RFC3339 approximate creation time of the stream record (empty for snapshot records) - `dynamodb_event_name` - The type of change: INSERT, MODIFY, REMOVE, or READ (for snapshot records) - `dynamodb_table` - The name of the DynamoDB table ## [](#metrics)Metrics This input emits the following metrics: - `dynamodb_cdc_shards_tracked`: Total number of shards being tracked (gauge) - `dynamodb_cdc_shards_active`: Number of shards currently being read from (gauge) ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.106.0 or later. **Type**: `bool` **Default**: `true` ### [](#batch_size)`batch_size` Maximum number of records to read per shard in a single request. Valid range: 1-1000. **Type**: `int` **Default**: `1000` ### [](#checkpoint_limit)`checkpoint_limit` Maximum number of unacknowledged messages before forcing a checkpoint update. Lower values provide better recovery guarantees but increase write overhead. **Type**: `int` **Default**: `1000` ### [](#checkpoint_namespace)`checkpoint_namespace` Isolates this pipeline’s checkpoints within a shared `checkpoint_table` by prefixing the namespace to the checkpoint key. Use this so that multiple pipelines reading the same stream can share one checkpoint table without overwriting each other’s positions, for example per-developer or per-environment test pipelines. Leave empty (the default) to keep the original checkpoint keys unchanged. A namespace isolates readers but does not coordinate them: pipelines that share the same namespace still collide. Changing or removing the namespace changes the checkpoint key. If no checkpoints exist yet under the new key, the pipeline starts from `start_from`. Switching back to a previously used namespace resumes from that namespace’s last checkpoints. The value cannot contain a `#` character. Requires version 4.101.0 or later. **Type**: `string` **Default**: `""` ### [](#checkpoint_table)`checkpoint_table` DynamoDB table name for storing checkpoints. Will be created if it doesn’t exist. **Type**: `string` **Default**: `redpanda_dynamodb_checkpoints` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#global_table)`global_table` Provision the checkpoint table as a DynamoDB Global Table (v2) so checkpoints replicate across regions. Requires `global_table_replicas`. When the table is auto-created it is created as a global table; when it already exists, its replicas are reconciled (missing regions are added by calling `UpdateTable`). The existing table must have been created in global mode (`TableId` hash key). Enabling this against a pre-existing non-global checkpoint table fails fast with a clear error. Requires version 4.99.0 or later. **Type**: `bool` **Default**: `false` ### [](#global_table_replicas)`global_table_replicas[]` Regions other than this pipeline’s own region to replicate the checkpoint table to. The pipeline’s own region is always included. Required when `global_table` is true. Applied both when the checkpoint table is created and, for an existing global table, when reconciling replicas (missing regions are added; this list is not used to remove regions). Requires version 4.99.0 or later. **Type**: `array` **Default**: `[]` ### [](#max_tracked_shards)`max_tracked_shards` Maximum number of shards to track simultaneously. Prevents memory issues with extremely large tables. **Type**: `int` **Default**: `10000` ### [](#poll_interval)`poll_interval` Time to wait between polling attempts when no records are available. **Type**: `string` **Default**: `1s` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#snapshot_batch_size)`snapshot_batch_size` Records per scan request during snapshot. Maximum 1000. Lower values provide better backpressure control but require more API calls. Requires version 4.82.0 or later. **Type**: `int` **Default**: `100` ### [](#snapshot_buffer_size)`snapshot_buffer_size` Maximum CDC events to buffer for deduplication (approximately 100 bytes per entry). If exceeded, deduplication is disabled and duplicates may be emitted. Requires version 4.82.0 or later. **Type**: `int` **Default**: `100000` ### [](#snapshot_deduplicate)`snapshot_deduplicate` Deduplicate records that appear in both snapshot and CDC stream. Requires buffering CDC events during snapshot. If buffer is exceeded, deduplication is disabled to prevent data loss. Requires version 4.82.0 or later. **Type**: `bool` **Default**: `true` ### [](#snapshot_mode)`snapshot_mode` `none`: Streams CDC events only (default). `snapshot_only`: Performs a one-time full table scan with no ongoing streaming. `snapshot_and_cdc`: Scans the entire table, then streams changes. Requires version 4.82.0 or later. **Type**: `string` **Default**: `none` **Options**: `none`, `snapshot_only`, `snapshot_and_cdc` ### [](#snapshot_segments)`snapshot_segments` Number of parallel scan segments (1-10). Higher parallelism scans faster but consumes more Read Capacity Units (RCUs). A lower value is safer to start with. Requires version 4.82.0 or later. **Type**: `int` **Default**: `1` ### [](#snapshot_throttle)`snapshot_throttle` Minimum time between scan requests per segment. Use this to limit Read Capacity Unit (RCU) consumption during snapshot. Requires version 4.82.0 or later. **Type**: `string` **Default**: `100ms` ### [](#start_from)`start_from` Where to start reading on a genuinely fresh pipeline (no checkpoint state exists yet under this `checkpoint_namespace` for the stream). `trim_horizon` starts from the oldest available record, `latest` starts from new records. `latest` is honoured only on that first discovery: once any checkpoint state exists, shards discovered later - rotation children found by the periodic refresh, and any checkpoint-less shard after a restart - always start at `trim_horizon` so their backlog is never skipped. In practice a restart under `latest` therefore replays from each shard’s oldest retained record rather than only new records; at-least-once delivery takes precedence over the configured start position. **Type**: `string` **Default**: `trim_horizon` **Options**: `trim_horizon`, `latest` ### [](#table_discovery_interval)`table_discovery_interval` Interval for rescanning and discovering new tables when using `tag` or `includelist` mode. Set to 0 to disable periodic rescanning. Requires version 4.83.0 or later. **Type**: `string` **Default**: `5m` ### [](#table_discovery_mode)`table_discovery_mode` `single`: Streams from tables specified in the `tables` list. `tag`: Auto-discovers tables by tags (ignores the `tables` field). `includelist`: Streams from tables in the `tables` list. Use `single` instead; `includelist` is kept for backward compatibility. Requires version 4.83.0 or later. **Type**: `string` **Default**: `single` **Options**: `single`, `tag`, `includelist` ### [](#table_tag_filter)`table_tag_filter` Multi-tag filter in the format `key1:v1,v2;key2:v3,v4`. Matches tables where (key1=v1 OR key1=v2) AND (key2=v3 OR key2=v4). Required when `table_discovery_mode` is `tag`. Requires version 4.83.0 or later. **Type**: `string` **Default**: `""` ### [](#tables)`tables[]` List of table names to stream from. For single table mode, provide one table. For multi-table mode, provide multiple tables. Requires version 4.83.0 or later. **Type**: `array` **Default**: `[]` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#throttle_backoff)`throttle_backoff` Time to wait when applying backpressure due to too many in-flight messages. **Type**: `string` **Default**: `100ms` ## [](#examples)Examples ### [](#consume-cdc-events)Consume CDC events Read change events from a DynamoDB table with streams enabled. ```yaml input: aws_dynamodb_cdc: tables: [my-table] region: us-east-1 ``` ### [](#start-from-latest)Start from latest Only process new changes, ignoring existing stream data. ```yaml input: aws_dynamodb_cdc: tables: [orders] start_from: latest region: us-west-2 ``` ### [](#snapshot-and-cdc)Snapshot and CDC Scan all existing records, then stream ongoing changes. ```yaml input: aws_dynamodb_cdc: tables: [products] snapshot_mode: snapshot_and_cdc snapshot_segments: 5 region: us-east-1 ``` ### [](#auto-discover-tables-by-tag)Auto-discover tables by tag Automatically discover and stream from all tables with a specific tag. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: tag table_tag_filter: "stream-enabled:true" table_discovery_interval: 5m region: us-east-1 ``` ### [](#auto-discover-tables-by-multiple-tags)Auto-discover tables by multiple tags Discover tables matching multiple tag criteria with OR logic per key, AND logic across keys. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: tag table_tag_filter: "environment:prod,staging;team:data,analytics" table_discovery_interval: 5m region: us-east-1 # Matches tables with: (environment=prod OR environment=staging) AND (team=data OR team=analytics) ``` ### [](#stream-from-multiple-specific-tables)Stream from multiple specific tables Stream from an explicit list of tables simultaneously. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: includelist tables: - orders - customers - products region: us-west-2 ``` ## [](#suggested-reading)Suggested reading For common patterns including filtering events, routing to Kafka or S3, and detecting changed fields, see the [DynamoDB CDC Patterns](https://docs.redpanda.com/connect/cookbooks/dynamodb_cdc/) cookbook. --- # Page 31: aws_kinesis **URL**: https://docs.redpanda.com/connect/components/inputs/aws_kinesis.md --- # aws_kinesis --- title: aws_kinesis latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/aws_kinesis page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_kinesis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_kinesis.adoc description: Receive messages from one or more Kinesis streams. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Receive messages from one or more Kinesis streams. Introduced in version 3.36.0. #### Common ```yml input: label: "" aws_kinesis: streams: [] # No default (required) dynamodb: table: "" create: false checkpoint_limit: 1024 auto_replay_nacks: true commit_period: 5s steal_grace_period: 2s start_from_oldest: true batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml input: label: "" aws_kinesis: streams: [] # No default (required) dynamodb: table: "" create: false billing_mode: PAY_PER_REQUEST read_capacity_units: 0 write_capacity_units: 0 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) checkpoint_limit: 1024 poll_period: 0s enhanced_fan_out: enabled: false consumer_name: "" consumer_activation_timeout: 1m max_resubscribe_interval: 30s auto_replay_nacks: true commit_period: 5s steal_grace_period: 2s rebalance_period: 30s lease_period: 30s start_from_oldest: true region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Consumes messages from one or more Kinesis streams either by automatically balancing shards across other instances of this input, or by consuming shards listed explicitly. The latest message sequence consumed by this input is stored within a [DynamoDB table](#table-schema), which allows it to resume at the correct sequence of the shard during restarts. This table is also used for coordination across distributed inputs when shard balancing. Redpanda Connect will not store a consumed sequence unless it is acknowledged at the output level, which ensures at-least-once delivery guarantees. ## [](#ordering)Ordering By default messages of a shard can be processed in parallel, up to a limit determined by the field `checkpoint_limit`. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level. ## [](#table-schema)Table schema It’s possible to configure Redpanda Connect to create the DynamoDB table required for coordination if it does not already exist. However, if you wish to create this yourself (recommended) then create a table with a string HASH key `StreamID` and a string RANGE key `ShardID`. ## [](#batching)Batching Use the `batching` fields to configure an optional [batching policy](https://docs.redpanda.com/connect/configuration/batching/#batch-policy). Each stream shard will be batched separately in order to ensure that acknowledgements aren’t contaminated. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum gap between the in flight sequence versus the latest acknowledged sequence at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual shards. Any given sequence will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#commit_period)`commit_period` The period of time between each update to the checkpoint table. **Type**: `string` **Default**: `5s` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#dynamodb)`dynamodb` Determines the table used for storing and accessing the latest consumed sequence for shards, and for coordinating balanced consumers of streams. **Type**: `object` ### [](#dynamodb-billing_mode)`dynamodb.billing_mode` When creating the table determines the billing mode. **Type**: `string` **Default**: `PAY_PER_REQUEST` **Options**: `PROVISIONED`, `PAY_PER_REQUEST` ### [](#dynamodb-create)`dynamodb.create` Whether, if the table does not exist, it should be created. **Type**: `bool` **Default**: `false` ### [](#dynamodb-credentials)`dynamodb.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). Requires version 4.53.0 or later. **Type**: `object` ### [](#dynamodb-credentials-from_ec2_role)`dynamodb.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#dynamodb-credentials-id)`dynamodb.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#dynamodb-credentials-profile)`dynamodb.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#dynamodb-credentials-role)`dynamodb.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#dynamodb-credentials-role_external_id)`dynamodb.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#dynamodb-credentials-secret)`dynamodb.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#dynamodb-credentials-token)`dynamodb.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#dynamodb-endpoint)`dynamodb.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. Requires version 4.53.0 or later. **Type**: `string` ### [](#dynamodb-read_capacity_units)`dynamodb.read_capacity_units` Set the provisioned read capacity when creating the table with a `billing_mode` of `PROVISIONED`. **Type**: `int` **Default**: `0` ### [](#dynamodb-region)`dynamodb.region` The AWS region in which your resources are hosted. Requires version 4.53.0 or later. **Type**: `string` ### [](#dynamodb-table)`dynamodb.table` The name of the table to access. **Type**: `string` **Default**: `""` ### [](#dynamodb-tcp)`dynamodb.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#dynamodb-tcp-connect_timeout)`dynamodb.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#dynamodb-tcp-keep_alive)`dynamodb.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#dynamodb-tcp-keep_alive-count)`dynamodb.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#dynamodb-tcp-keep_alive-idle)`dynamodb.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#dynamodb-tcp-keep_alive-interval)`dynamodb.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#dynamodb-tcp-tcp_user_timeout)`dynamodb.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#dynamodb-write_capacity_units)`dynamodb.write_capacity_units` Set the provisioned write capacity when creating the table with a `billing_mode` of `PROVISIONED`. **Type**: `int` **Default**: `0` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#enhanced_fan_out)`enhanced_fan_out` Consume the stream using [enhanced fan-out](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html), which provides this consumer dedicated read throughput of 2MB/s per shard via HTTP/2 push delivery, avoiding the 5 reads per second per shard limit that polling consumers share. The named consumer is registered on each stream automatically if it does not already exist (and is never deregistered). Requires the IAM permissions `kinesis:DescribeStreamConsumer` and `kinesis:SubscribeToShard`, plus `kinesis:RegisterStreamConsumer` unless a consumer with the configured name already exists on every stream (for example one provisioned through infrastructure-as-code). Note that AWS bills enhanced fan-out consumers per consumer-shard-hour plus data retrieval. When a shard has no stored checkpoint and `start_from_oldest` is `false`, consumption starts from a timestamp anchored to this host’s clock at claim time (rather than the server-side `LATEST` used by polling, which cannot be pinned across resubscribes without risking skipped records), so a clock running ahead can omit its offset’s worth of records after a fresh claim. Requires version 4.109.0 or later. **Type**: `object` ### [](#enhanced_fan_out-consumer_activation_timeout)`enhanced_fan_out.consumer_activation_timeout` The maximum amount of time to wait on connect for the registered consumer to become active before failing. Newly registered consumers on streams with many shards can take tens of seconds to activate. **Type**: `string` **Default**: `1m` ### [](#enhanced_fan_out-consumer_name)`enhanced_fan_out.consumer_name` The name of the enhanced fan-out consumer to register. Required when `enabled` is true. Each distinct pipeline (application) consuming a stream must use its own consumer name, as Kinesis permits only one active subscription per consumer per shard. Instances of the same pipeline sharing a DynamoDB checkpoint table should share this name. **Type**: `string` **Default**: `""` ### [](#enhanced_fan_out-enabled)`enhanced_fan_out.enabled` Whether to consume the stream using enhanced fan-out. **Type**: `bool` **Default**: `false` ### [](#enhanced_fan_out-max_resubscribe_interval)`enhanced_fan_out.max_resubscribe_interval` The ceiling on the exponential backoff between SubscribeToShard attempts after a subscription ends without delivering any events. This bounds how long a shard may sit unsubscribed after repeated failures. **Type**: `string` **Default**: `30s` ### [](#lease_period)`lease_period` The period of time after which a client that has failed to update a shard checkpoint is assumed to be inactive. **Type**: `string` **Default**: `30s` ### [](#poll_period)`poll_period` An optional minimum period between GetRecords calls made against each shard. Kinesis allows a shared budget of 5 GetRecords calls per second per shard across all consumers of a stream, so a value such as `250ms` bounds this consumer to roughly four reads per second per shard, leaving headroom for other consumers of the same stream. The default of `0s` polls as fast as records are consumed. This setting has no effect when `enhanced_fan_out` is enabled. A shard is polled at most once per period, so the committed sequence advances no faster than that; values above `lease_period` are rejected. Requires version 4.109.0 or later. **Type**: `string` **Default**: `0s` ### [](#rebalance_period)`rebalance_period` The period of time between each attempt to rebalance shards across clients. **Type**: `string` **Default**: `30s` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#start_from_oldest)`start_from_oldest` Whether to consume from the oldest message when a sequence does not yet exist for the stream. **Type**: `bool` **Default**: `true` ### [](#steal_grace_period)`steal_grace_period` Determines how long beyond the next commit period a client will wait when stealing a shard for the current owner to store a checkpoint. A longer value increases the time taken to balance shards but reduces the likelihood of processing duplicate messages. Requires version 4.55.0 or later. **Type**: `string` **Default**: `2s` ### [](#streams)`streams[]` One or more Kinesis data streams to consume from. Streams can either be specified by their name or full ARN. Shards of a stream are automatically balanced across consumers by coordinating through the provided DynamoDB table. Multiple comma separated streams can be listed in a single element. Shards are automatically distributed across consumers of a stream by coordinating through the provided DynamoDB table. Alternatively, it’s possible to specify an explicit shard to consume from with a colon after the stream name. For example `foo:0` would consume the shard `0` of the stream `foo`. **Type**: `array` ```yaml # Examples: streams: - foo - "arn:aws:kinesis:*:111122223333:stream/my-stream" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 32: aws_s3 **URL**: https://docs.redpanda.com/connect/components/inputs/aws_s3.md --- # aws_s3 --- title: aws_s3 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/aws_s3 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_s3.adoc description: Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifications in realtime. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifications in real time. #### Common ```yml input: label: "" aws_s3: bucket: "" prefix: "" scanner: to_the_end: {} sqs: url: "" key_path: Records.*.s3.object.key bucket_path: Records.*.s3.bucket.name envelope_path: "" nack_visibility_timeout: 0 ``` #### Advanced ```yml input: label: "" aws_s3: bucket: "" prefix: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) force_path_style_urls: false delete_objects: false scanner: to_the_end: {} sqs: url: "" endpoint: "" key_path: Records.*.s3.object.key bucket_path: Records.*.s3.bucket.name envelope_path: "" delay_period: "" max_messages: 10 wait_time_seconds: 0 nack_visibility_timeout: 0 zero_key_warn_interval: 30s ``` ## [](#stream-objects-on-upload-with-sqs)Stream objects on upload with SQS A common pattern for consuming S3 objects is to emit upload notification events from the bucket either directly to an SQS queue, or to an SNS topic that is consumed by an SQS queue, and then have your consumer listen for events that prompt it to download the newly uploaded objects. More information about this pattern and how to set it up can be found in the [Amazon S3 docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html). Redpanda Connect is able to follow this pattern when you configure an `sqs.url`, where it consumes events from SQS and downloads only the object keys contained in those events. For this to work, Redpanda Connect needs to know where within the event the key and bucket names can be found, specified as [dot paths](https://docs.redpanda.com/connect/configuration/field_paths/) with the fields `sqs.key_path` and `sqs.bucket_path`. The default values for these fields should already be correct when following the guide above. If your notification events are being routed to SQS via an SNS topic, the events are enveloped by SNS, in which case you also need to specify the field `sqs.envelope_path`, which in the case of SNS to SQS will usually be `Message`. When using SQS, make sure you have sensible values for `sqs.max_messages` and also the visibility timeout of the queue itself. When Redpanda Connect consumes an S3 object the SQS message that triggered it is not deleted until the S3 object has been sent onwards. This ensures at-least-once crash resiliency, but also means that if the S3 object takes longer to process than the visibility timeout of your queue, then the same objects might be processed multiple times. ## [](#download-large-files)Download large files When downloading large files, process them in streamed parts to avoid loading the entire file into memory at once. To do this, specify a [`scanner`](#scanner) that determines how to break the input into smaller individual messages. ## [](#bucket-and-prefix)Bucket and prefix The `bucket` field accepts a bucket name only, not an ARN. For example, use `my-bucket`, not `arn:aws:s3:::my-bucket`. The `prefix` field accepts a single string. To consume from multiple prefixes in the same bucket, use multiple `aws_s3` inputs in a [`broker` input](https://docs.redpanda.com/connect/components/inputs/broker/): ```yaml input: broker: inputs: - aws_s3: bucket: my-bucket prefix: logs/app1/ - aws_s3: bucket: my-bucket prefix: logs/app2/ ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#s3-compatible-storage)S3-compatible storage The `endpoint` and `force_path_style_urls` fields let you connect to S3-compatible storage services such as Cloudflare R2, MinIO, or DigitalOcean Spaces. For Cloudflare R2, set `endpoint` to your account endpoint URL and enable `force_path_style_urls`: ```yaml input: aws_s3: bucket: r2-bucket endpoint: https://.r2.cloudflarestorage.com force_path_style_urls: true region: auto credentials: id: secret: ``` Find your account ID in the Cloudflare dashboard under **R2 > Overview > Account Details**. Generate API credentials under **R2 > Manage R2 API Tokens**. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `s3_key` - `s3_bucket` - `s3_last_modified_unix` - `s3_last_modified` (RFC3339) - `s3_content_type` - `s3_content_encoding` - `s3_version_id` - All user defined metadata You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Note that user defined metadata is case insensitive within AWS, and it is likely that the keys will be received in a capitalized form, if you wish to make them consistent you can map all metadata keys to lower or uppercase using a Bloblang mapping such as `meta = meta().map_each_key(key → key.lowercase())`. ## [](#fields)Fields ### [](#bucket)`bucket` The name of the bucket to consume from, such as `my-bucket`. Use the bucket name, not its ARN. If the field `sqs.url` is specified this field is optional. **Type**: `string` **Default**: `""` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#delete_objects)`delete_objects` Whether to delete each object from its S3 bucket after all of its messages are acknowledged successfully. Objects whose messages fail to process are left in the bucket. **Type**: `bool` **Default**: `false` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs for downloading keys, which is often required when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` An optional path prefix, if set only objects with the prefix are consumed when walking a bucket. This field accepts a single prefix. To consume objects under several prefixes in the same bucket, use one `aws_s3` input for each prefix in a [`broker` input](https://docs.redpanda.com/connect/components/inputs/broker/). **Type**: `string` **Default**: `""` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#sqs)`sqs` Consume SQS messages in order to trigger key downloads. **Type**: `object` ### [](#sqs-bucket_path)`sqs.bucket_path` A [dot path](https://docs.redpanda.com/connect/configuration/field_paths/) whereby the bucket name can be found in SQS messages. **Type**: `string` **Default**: `Records.*.s3.bucket.name` ### [](#sqs-delay_period)`sqs.delay_period` An optional period of time to wait from when a notification was originally sent to when the target key download is attempted. **Type**: `string` **Default**: `""` ```yaml # Examples: delay_period: 10s # --- delay_period: 5m ``` ### [](#sqs-endpoint)`sqs.endpoint` A custom endpoint to use when connecting to SQS. **Type**: `string` **Default**: `""` ### [](#sqs-envelope_path)`sqs.envelope_path` A [dot path](https://docs.redpanda.com/connect/configuration/field_paths/) of a field to extract an enveloped JSON payload for further extracting the key and bucket from SQS messages. This is specifically useful when subscribing an SQS queue to an SNS topic that receives bucket events. **Type**: `string` **Default**: `""` ```yaml # Examples: envelope_path: Message ``` ### [](#sqs-key_path)`sqs.key_path` A [dot path](https://docs.redpanda.com/connect/configuration/field_paths/) whereby object keys are found in SQS messages. **Type**: `string` **Default**: `Records.*.s3.object.key` ### [](#sqs-max_messages)`sqs.max_messages` The maximum number of SQS messages to consume from each request. **Type**: `int` **Default**: `10` ### [](#sqs-nack_visibility_timeout)`sqs.nack_visibility_timeout` Custom SQS Nack Visibility timeout in seconds. Default is 0 Requires version 4.78.0 or later. **Type**: `int` **Default**: `0` ### [](#sqs-url)`sqs.url` An optional SQS URL to connect to. When specified this queue will control which objects are downloaded. **Type**: `string` **Default**: `""` ### [](#sqs-wait_time_seconds)`sqs.wait_time_seconds` Whether to set the wait time. Enabling this activates long-polling. Valid values: 0 to 20. Requires version 4.17.0 or later. **Type**: `int` **Default**: `0` ### [](#sqs-zero_key_warn_interval)`sqs.zero_key_warn_interval` A message from which no target key can be extracted (for example, due to a misconfigured `key_path`/`bucket_path`) is never deleted, so it’s redelivered and re-evaluated repeatedly until the underlying issue is fixed. This field limits how often that condition is logged as a warning, to avoid flooding the logs; every occurrence is still logged at debug level. Set to `0s` to warn on every occurrence. Requires version 4.103.2 or later. **Type**: `string` **Default**: `30s` ```yaml # Examples: zero_key_warn_interval: 10s # --- zero_key_warn_interval: 0s ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 33: aws_sqs **URL**: https://docs.redpanda.com/connect/components/inputs/aws_sqs.md --- # aws_sqs --- title: aws_sqs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/aws_sqs page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_sqs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_sqs.adoc description: Consume messages from an AWS SQS URL. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consume messages from an AWS SQS URL. #### Common ```yml input: label: "" aws_sqs: url: "" # No default (required) max_outstanding_messages: 1000 ``` #### Advanced ```yml input: label: "" aws_sqs: url: "" # No default (required) delete_message: true reset_visibility: true max_number_of_messages: 10 max_outstanding_messages: 1000 wait_time_seconds: 0 message_timeout: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level, which allows you to transfer data across accounts. To find out more, see [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `sqs_message_id` - `sqs_receipt_handle` - `sqs_approximate_receive_count` - All message attributes You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#delete_message)`delete_message` Whether to delete the consumed message when it’s acknowledged. Set to `false` to handle the deletion using a different mechanism. **Type**: `bool` **Default**: `true` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_number_of_messages)`max_number_of_messages` The maximum number of messages that Redpanda Connect can return each time it polls the SQS queue. Enter a value from `1` to `10`. **Type**: `int` **Default**: `10` ### [](#max_outstanding_messages)`max_outstanding_messages` The maximum number of pending messages that Redpanda Connect can have in flight at the same time. Requires version 4.45.0 or later. **Type**: `int` **Default**: `1000` ### [](#message_timeout)`message_timeout` The maximum time allowed to process a received message before Redpanda Connect must refresh its [receipt handle](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-message-identifiers.html). If the handle is not refreshed in time, the message becomes visible in the queue again. Redpanda Connect attempts to refresh the receipt handle after half of the timeout has elapsed. This value sets the visibility timeout for each received message. Requires version 4.45.0 or later. **Type**: `string` **Default**: `30s` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#reset_visibility)`reset_visibility` Whether to set the visibility timeout of the consumed message to zero when Redpanda Connect receives a negative acknowledgement, which releases the message immediately for reprocessing. Set to `false` to honor the [queue’s visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) instead. Requires version 3.58.0 or later. **Type**: `bool` **Default**: `true` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#url)`url` The SQS URL to consume from. **Type**: `string` ### [](#wait_time_seconds)`wait_time_seconds` The wait time, in seconds, for each receive request. Valid values are `0` to `20`. Set a value from `1` to `20` to enable [long polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) for queued messages. Requires version 4.6.0 or later. **Type**: `int` **Default**: `0` --- # Page 34: azure_blob_storage **URL**: https://docs.redpanda.com/connect/components/inputs/azure_blob_storage.md --- # azure_blob_storage --- title: azure_blob_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/azure_blob_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_blob_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_blob_storage.adoc description: Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix. Introduced in version 3.36.0. Supports multiple authentication methods but only one of the following is required: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access via [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If multiple are set then the `storage_connection_string` is given priority. If the `storage_connection_string` does not contain the `AccountName` parameter, please specify it in the `storage_account` field. ## [](#download-large-files)Download large files When downloading large files it’s often necessary to process it in streamed parts in order to avoid loading the entire file in memory at a given time. In order to do this a [`scanner`](#scanner) can be specified that determines how to break the input into smaller individual messages. ## [](#stream-new-files)Stream new files By default this input will consume all files found within the target container and will then gracefully terminate. This is referred to as a "batch" mode of operation. However, it’s possible to instead configure a container as [an Event Grid source](https://learn.microsoft.com/en-gb/azure/event-grid/event-schema-blob-storage) and then use this as a [`targets_input`](#targets_input), in which case new files are consumed as they’re uploaded and Redpanda Connect will continue listening for and downloading files as they arrive. This is referred to as a "streamed" mode of operation. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `blob_storage_key` - `blob_storage_container` - `blob_storage_last_modified` - `blob_storage_last_modified_unix` - `blob_storage_content_type` - `blob_storage_content_encoding` - All user defined metadata You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) prefix: "" scanner: to_the_end: {} targets_input: {} # No default (optional) ``` ### Advanced ```yml input: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) prefix: "" scanner: to_the_end: {} delete_objects: false targets_input: {} # No default (optional) ``` ## [](#fields)Fields ### [](#container)`container` The name of the container from which to download blobs. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#delete_objects)`delete_objects` Whether to delete downloaded blobs from the container once they are processed. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` An optional path prefix, if set only objects with the prefix are consumed. **Type**: `string` **Default**: `""` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` ### [](#targets_input)`targets_input` > ⚠️ **CAUTION** > > This is an experimental field that provides an optional source of download targets, configured as a [regular Redpanda Connect input](https://docs.redpanda.com/connect/components/inputs/about/). Each message yielded by this input should be a single structured object containing a field `name`, which represents the blob to be downloaded. This requires setting up [Azure Blob Storage as an Event Grid source](https://learn.microsoft.com/en-gb/azure/event-grid/event-schema-blob-storage) and an associated event handler that a Redpanda Connect input can read from. For example, use either one of the following: - [Azure Event Hubs](https://learn.microsoft.com/en-gb/azure/event-grid/handler-event-hubs) using the `kafka` input - [Namespace topics](https://learn.microsoft.com/en-gb/azure/event-grid/handler-event-grid-namespace-topic) using the `mqtt` input Requires version 4.27.0 or later. **Type**: `input` ```yaml # Examples: targets_input: mqtt: topics: - some-topic urls: - example.westeurope-1.ts.eventgrid.azure.net:8883 processors: - unarchive: format: json_array - mapping: |- if this.eventType == "Microsoft.Storage.BlobCreated" { root.name = this.data.url.parse_url().path.trim_prefix("/foocontainer/") } else { root = deleted() } ``` --- # Page 35: azure_cosmosdb **URL**: https://docs.redpanda.com/connect/components/inputs/azure_cosmosdb.md --- # azure_cosmosdb --- title: azure_cosmosdb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/azure_cosmosdb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_cosmosdb.adoc description: Executes a SQL query against Azure CosmosDB and creates a batch of messages from each page of items. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a SQL query against [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction) and creates a batch of messages from each page of items. Introduced in version 4.25.0. ## [](#cross-partition-queries)Cross-partition queries Cross-partition queries are currently not supported by the underlying driver. For every query, the PartitionKey values must be known in advance and specified in the config. [See details](https://github.com/Azure/azure-sdk-for-go/issues/18578#issuecomment-1222510989). ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#metadata)Metadata This component adds the following metadata fields to each message: - `activity_id` - `request_charge` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) batch_count: -1 auto_replay_nacks: true ``` ## [](#examples)Examples ### [](#query-container)Query container Execute a parametrized SQL query to select documents from a container. ```yaml input: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = "AbyssalPlain" query: SELECT * FROM blobfish AS b WHERE b.species = @species args_mapping: | root = [ { "Name": "@species", "Value": "smooth-head" }, ] ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that creates the list of parameters to use with the query. Each parameter is an object with a `Name` field, which matches a named parameter in the query such as `@name`, and a `Value` field. The input evaluates this mapping once, when the input is created, so the mapping cannot reference message contents or metadata. **Type**: `string` ```yaml # Examples: args_mapping: |- root = [ { "Name": "@name", "Value": "benthos" }, ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` The maximum number of messages that should be accumulated into each batch. Use '-1' specify dynamic page size. **Type**: `int` **Default**: `-1` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = now().ts_format("2006-01-02") ``` ### [](#query)`query` The query to execute **Type**: `string` ```yaml # Examples: query: SELECT c.foo FROM testcontainer AS c WHERE c.bar = "baz" AND c.timestamp < @timestamp ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 36: azure_queue_storage **URL**: https://docs.redpanda.com/connect/components/inputs/azure_queue_storage.md --- # azure_queue_storage --- title: azure_queue_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/azure_queue_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_queue_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_queue_storage.adoc description: Dequeue objects from an Azure Storage Queue. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Dequeue objects from an Azure Storage Queue. Introduced in version 3.42.0. This input adds the following metadata fields to each message: ```none - queue_storage_insertion_time - queue_storage_queue_name - queue_storage_message_lag (if 'track_properties' set to true) - All user defined queue metadata ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. #### Common ```yml input: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" queue_name: "" # No default (required) ``` #### Advanced ```yml input: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" queue_name: "" # No default (required) dequeue_visibility_timeout: 30s max_in_flight: 10 track_properties: false ``` ## [](#fields)Fields ### [](#dequeue_visibility_timeout)`dequeue_visibility_timeout` The timeout duration until a dequeued message gets visible again, 30s by default Requires version 3.45.0 or later. **Type**: `string` **Default**: `30s` ### [](#max_in_flight)`max_in_flight` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `10` ### [](#queue_name)`queue_name` The name of the source storage queue. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: queue_name: foo_queue ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#track_properties)`track_properties` If set to `true` the queue is polled on each read request for information such as the queue message lag. These properties are added to consumed messages as metadata, but will also have a negative performance impact. **Type**: `bool` **Default**: `false` --- # Page 37: azure_table_storage **URL**: https://docs.redpanda.com/connect/components/inputs/azure_table_storage.md --- # azure_table_storage --- title: azure_table_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/azure_table_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_table_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_table_storage.adoc description: Queries an Azure Storage Account Table, optionally with multiple filters. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Queries an Azure Storage Account Table, optionally with multiple filters. Introduced in version 4.10.0. Queries an Azure Storage Account Table, optionally with multiple filters. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `table_storage_name` - `row_num` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) ``` ### Advanced ```yml input: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) filter: "" select: "" page_size: 1000 ``` ## [](#fields)Fields ### [](#filter)`filter` OData filter expression. Is not set all rows are returned. Valid operators are `eq, ne, gt, lt, ge and le` **Type**: `string` **Default**: `""` ```yaml # Examples: filter: PartitionKey eq 'foo' and RowKey gt '1000' ``` ### [](#page_size)`page_size` Maximum number of records to return on each page. **Type**: `int` **Default**: `1000` ### [](#select)`select` Select expression using OData notation. Limits the columns on each record to just those requested. **Type**: `string` **Default**: `""` ```yaml # Examples: select: PartitionKey,RowKey,Foo,Bar,Timestamp ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` ### [](#table_name)`table_name` The table to read messages from. **Type**: `string` ```yaml # Examples: table_name: Foo ``` --- # Page 38: batched **URL**: https://docs.redpanda.com/connect/components/inputs/batched.md --- # batched --- title: batched latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/batched page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/batched.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/batched.adoc description: Consumes data from a child input and applies a batching policy to the stream. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes data from a child input and applies a batching policy to the stream. Introduced in version 4.11.0. Batching at the input level is sometimes useful for processing across micro-batches, and can also sometimes be a useful performance trick. However, most inputs are fine without it so unless you have a specific plan for batching this component is not worth using. #### Common ```yml input: label: "" batched: child: {} # No default (required) policy: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml input: label: "" batched: child: {} # No default (required) policy: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#child)`child` The child input. **Type**: `input` ### [](#policy)`policy` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: policy: byte_size: 5000 count: 0 period: 1s # --- policy: count: 10 period: 1s # --- policy: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#policy-byte_size)`policy.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#policy-check)`policy.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#policy-count)`policy.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#policy-period)`policy.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#policy-processors)`policy.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` --- # Page 39: beanstalkd **URL**: https://docs.redpanda.com/connect/components/inputs/beanstalkd.md --- # beanstalkd --- title: beanstalkd latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/beanstalkd page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/beanstalkd.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/beanstalkd.adoc description: Reads messages from a Beanstalkd queue. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads messages from a Beanstalkd queue. Introduced in version 4.7.0. #### Common ```yml input: label: "" beanstalkd: address: "" # No default (required) ``` #### Advanced ```yml input: label: "" beanstalkd: address: "" # No default (required) ``` ## [](#fields)Fields ### [](#address)`address` An address to connect to. **Type**: `string` ```yaml # Examples: address: 127.0.0.1:11300 ``` --- # Page 40: broker **URL**: https://docs.redpanda.com/connect/components/inputs/broker.md --- # broker --- title: broker latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/broker page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/broker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/broker.adoc description: Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel. A broker type is configured with its own list of input configurations and a field to specify how many copies of the list of inputs should be created. Adding more input types allows you to combine streams from multiple sources into one. For example, reading from both RabbitMQ and Kafka: ```yaml input: broker: copies: 1 inputs: - amqp_0_9: urls: - amqp://guest:guest@localhost:5672/ consumer_tag: benthos-consumer queue: benthos-queue # Optional list of input specific processing steps processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() - kafka: addresses: - localhost:9092 client_id: benthos_kafka_input consumer_group: benthos_consumer_group topics: [ benthos_stream:0 ] ``` If the number of copies is greater than zero the list will be copied that number of times. For example, if your inputs were of type foo and bar, with 'copies' set to '2', you would end up with two 'foo' inputs and two 'bar' inputs. ## [](#batching)Batching It’s possible to configure a [batch policy](https://docs.redpanda.com/connect/configuration/batching/#batch-policy) with a broker using the `batching` fields. When doing this the feeds from all child inputs are combined. Some inputs do not support broker based batching and specify this in their documentation. ## [](#processors)Processors It is possible to configure [processors](https://docs.redpanda.com/connect/components/processors/about/) at the broker level, where they will be applied to _all_ child inputs, as well as on the individual child inputs. If you have processors at both the broker level _and_ on child inputs then the broker processors will be applied _after_ the child nodes processors. ### Common ```yml input: label: "" broker: inputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml input: label: "" broker: copies: 1 inputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#copies)`copies` Whatever is specified within `inputs` will be created this many times. **Type**: `int` **Default**: `1` ### [](#inputs)`inputs[]` A list of inputs to create. **Type**: `array` --- # Page 41: cassandra **URL**: https://docs.redpanda.com/connect/components/inputs/cassandra.md --- # cassandra --- title: cassandra latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/cassandra page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/cassandra.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/cassandra.adoc description: Executes a find query and creates a message for each row received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a find query and creates a message for each row received. Introduced in version 4.10.0. #### Common ```yml input: label: "" cassandra: addresses: [] # No default (required) timeout: 600ms reconnect_interval: 60s query: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" cassandra: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] password_authenticator: enabled: false username: "" password: "" disable_initial_host_lookup: false max_retries: 3 backoff: initial_interval: 1s max_interval: 5s timeout: 600ms host_selection_policy: local_dc: "" # No default (optional) local_rack: "" # No default (optional) reconnect_interval: 60s exponential_reconnection: max_retries: 0 # No default (required) initial_interval: "" # No default (required) max_interval: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true ``` ## [](#examples)Examples ### [](#minimal-select-cassandrascylla)Minimal Select (Cassandra/Scylla) Let’s presume that we have 3 Cassandra nodes, like in this tutorial by Sebastian Sigl from freeCodeCamp: [https://www.freecodecamp.org/news/the-apache-cassandra-beginner-tutorial/](https://www.freecodecamp.org/news/the-apache-cassandra-beginner-tutorial/) Then if we want to select everything from the table users\_by\_country, we should use the configuration below. If we specify the stdin output, the result will look like: ```json {"age":23,"country":"UK","first_name":"Bob","last_name":"Sandler","user_email":"bob@email.com"} ``` This configuration also works for Scylla. ```yaml input: cassandra: addresses: - 172.17.0.2 query: 'SELECT * FROM learn_cassandra.users_by_country' ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of Cassandra nodes to connect to. Multiple comma separated addresses can be specified on a single line. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9042" # --- addresses: - "foo:9042" - "bar:9042" # --- addresses: - "foo:9042,bar:9042" ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#backoff)`backoff` The backoff the Cassandra driver applies between retries of a failed query. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The period to wait before the first retry of a failed query. Each later retry doubles the previous wait, with random jitter of up to half this value, until `backoff.max_interval` is reached. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` The longest wait between retries of a failed query. **Type**: `string` **Default**: `5s` ### [](#disable_initial_host_lookup)`disable_initial_host_lookup` If enabled the driver will not attempt to get host info from the system.peers table. This can speed up queries but will mean that data\_centre, rack and token information will not be available. **Type**: `bool` **Default**: `false` ### [](#exponential_reconnection)`exponential_reconnection` Exponential backoff between the connection attempts that the driver makes each time it connects to a host. It applies on the initial connection and on each `reconnect_interval` attempt to a node marked as DOWN, and replaces the driver’s default of 3 attempts 1s apart. Requires version 4.66.0 or later. **Type**: `object` ### [](#exponential_reconnection-initial_interval)`exponential_reconnection.initial_interval` The base wait between the connection attempts the driver makes each time it connects to a host. The first wait is a random duration up to this value, and each later wait doubles, with random jitter of plus or minus half this value, until `max_interval` is reached. **Type**: `string` ### [](#exponential_reconnection-max_interval)`exponential_reconnection.max_interval` The longest wait between the connection attempts each time the driver connects to a host. **Type**: `string` ### [](#exponential_reconnection-max_retries)`exponential_reconnection.max_retries` The maximum number of connection attempts each time the driver connects to a host. **Type**: `int` ### [](#host_selection_policy)`host_selection_policy` Advanced host selection policy settings for Cassandra clusters, highly recommended in multi-datacenter (DC) deployments. Use these options to optimize query routing in multi-DC and multi-rack deployments. By specifying a local DC and rack, you can use the DC-aware and rack-aware policies to direct queries to the closest nodes, reducing latency and improving fault tolerance. If not set, the default policy is round-robin across all available nodes. Host selection is always token-aware if the token can be calculated from the query. Requires version 4.61.0 or later. **Type**: `object` ```yaml # Examples: host_selection_policy: local_dc: dc-east local_rack: rack1 ``` ### [](#host_selection_policy-local_dc)`host_selection_policy.local_dc` The name of the local datacenter to prioritize for query routing. Enables DC-aware host selection, ensuring queries are sent to nodes within this datacenter whenever possible. Recommended for clusters spanning multiple datacenters to minimize cross-DC traffic. **Type**: `string` ### [](#host_selection_policy-local_rack)`host_selection_policy.local_rack` The name of the local rack to prioritize for query routing. Requires `local_dc` to be set. Enables rack-aware host selection, further optimizing query placement within the specified datacenter. Useful for deployments with multiple racks per datacenter to improve resilience and reduce intra-DC latency. **Type**: `string` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on a request. **Type**: `int` **Default**: `3` ### [](#password_authenticator)`password_authenticator` Optional configuration of Cassandra authentication parameters. **Type**: `object` ### [](#password_authenticator-enabled)`password_authenticator.enabled` Whether to use password authentication **Type**: `bool` **Default**: `false` ### [](#password_authenticator-password)`password_authenticator.password` The password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#password_authenticator-username)`password_authenticator.username` The username to authenticate as. **Type**: `string` **Default**: `""` ### [](#query)`query` A query to execute. **Type**: `string` ### [](#reconnect_interval)`reconnect_interval` The interval at which Redpanda Connect attempts to reconnect to Cassandra nodes that are marked as DOWN. This setting helps maintain connectivity in unstable network conditions or during node maintenance. Use Go duration format such as `30s`, `1m`, or `5m`. Setting this too low may create unnecessary connection attempts, while setting it too high may delay recovery from network issues. Requires version 4.66.0 or later. **Type**: `string` **Default**: `60s` ### [](#timeout)`timeout` The client connection timeout. **Type**: `string` **Default**: `600ms` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. Requires version 4.25.0 or later. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 42: cockroachdb_changefeed **URL**: https://docs.redpanda.com/connect/components/inputs/cockroachdb_changefeed.md --- # cockroachdb_changefeed --- title: cockroachdb_changefeed latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/cockroachdb_changefeed page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/cockroachdb_changefeed.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/cockroachdb_changefeed.adoc description: Listens to a CockroachDB Core Changefeed and creates a message for each row received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Listens to a [CockroachDB Core Changefeed](https://www.cockroachlabs.com/docs/stable/changefeed-examples) and creates a message for each row received. Each message is a json object looking like: ```json { "primary_key": "[\"1a7ff641-3e3b-47ee-94fe-a0cadb56cd8f\", 2]", // stringifed JSON array "row": "{\"after\": {\"k\": \"1a7ff641-3e3b-47ee-94fe-a0cadb56cd8f\", \"v\": 2}, \"updated\": \"1637953249519902405.0000000000\"}", // stringified JSON object "table": "strm_2" } ``` #### Common ```yml input: label: "" cockroachdb_changefeed: dsn: "" # No default (required) tables: [] # No default (required) cursor_cache: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" cockroachdb_changefeed: dsn: "" # No default (required) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tables: [] # No default (required) cursor_cache: "" # No default (optional) options: [] # No default (optional) auto_replay_nacks: true ``` This input will continue to listen to the changefeed until shutdown. A backfill of the full current state of the table will be delivered upon each run unless a cache is configured for storing cursor timestamps, as this is how Redpanda Connect keeps track as to which changes have been successfully delivered. Note: You must have `SET CLUSTER SETTING kv.rangefeed.enabled = true;` on your CRDB cluster, for more information refer to [the official CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/changefeed-examples?filters=core). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#cursor_cache)`cursor_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) to use for storing the latest cursor that has been successfully delivered. This allows Redpanda Connect to continue from that cursor upon restart, rather than consume the entire state of the table. **Type**: `string` ### [](#dsn)`dsn` A Data Source Name to identify the target database. **Type**: `string` ```yaml # Examples: dsn: postgres://user:password@example.com:26257/defaultdb?sslmode=require ``` ### [](#options)`options[]` A list of options to be included in the changefeed (WITH X, Y…​). > 📝 **NOTE** > > Both the CURSOR option and UPDATED will be ignored from these options when a `cursor_cache` is specified, as they are set explicitly by Redpanda Connect in this case. A RESOLVED option is also added (unless one is supplied here): the stored cursor only ever advances to resolved timestamps whose rows have all been acknowledged downstream, so a restart redelivers at most the changes since the last resolved timestamp (bounded by the `changefeed.min_checkpoint_frequency` cluster setting) and never skips data. Resolved records are internal cursor bookkeeping and are never emitted as messages; without a `cursor_cache` they are discarded. **Type**: `array` ```yaml # Examples: options: - "virtual_columns=\"omitted\"" ``` ### [](#tables)`tables[]` CSV of tables to be included in the changefeed **Type**: `array` ```yaml # Examples: tables: - table1 - table2 ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 43: csv **URL**: https://docs.redpanda.com/connect/components/inputs/csv.md --- # csv --- title: csv latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/csv page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/csv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/csv.adoc description: Reads one or more CSV files as structured records following the format described in RFC 4180. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads one or more CSV files as structured records following the format described in RFC 4180. This input offers more control over CSV parsing than the [`file` input](https://docs.redpanda.com/connect/components/inputs/file/). When parsing with a header row each line of the file will be consumed as a structured object, where the key names are determined from the header now. For example, the following CSV file: ```csv foo,bar,baz first foo,first bar,first baz second foo,second bar,second baz ``` Would produce the following messages: ```json {"foo":"first foo","bar":"first bar","baz":"first baz"} {"foo":"second foo","bar":"second bar","baz":"second baz"} ``` If, however, the field `parse_header_row` is set to `false` then arrays are produced instead, like follows: ```json ["first foo","first bar","first baz"] ["second foo","second bar","second baz"] ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `header` - `path` - `mod_time_unix` - `mod_time` (RFC3339) You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Note: The `header` field is only set when `parse_header_row` is `true`. ## [](#output-csv-column-order)Output CSV column order When [creating CSV](https://docs.redpanda.com/connect/guides/bloblang/advanced/#creating-csv) from Redpanda Connect messages, the columns must be sorted lexicographically to make the output deterministic. Alternatively, when using the `csv` input, one can leverage the `header` metadata field to retrieve the column order: ```yaml input: csv: paths: - ./foo.csv - ./bar.csv parse_header_row: true processors: - mapping: | map escape_csv { root = if this.re_match("[\"\n,]+") { "\"" + this.replace_all("\"", "\"\"") + "\"" } else { this } } let header = if count(@path) == 1 { @header.map_each(c -> c.apply("escape_csv")).join(",") + "\n" } else { "" } root = $header + @header.map_each(c -> this.get(c).string().apply("escape_csv")).join(",") output: file: path: ./output/${! @path.filepath_split().index(-1) } ``` ### Common ```yml input: label: "" csv: paths: [] # No default (required) parse_header_row: true delimiter: "," lazy_quotes: false auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" csv: paths: [] # No default (required) parse_header_row: true delimiter: "," lazy_quotes: false delete_on_finish: false batch_count: 1 auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` Optionally process records in batches. This can help to speed up the consumption of exceptionally large CSV files. When the end of the file is reached the remaining records are processed as a (potentially smaller) batch. **Type**: `int` **Default**: `1` ### [](#delete_on_finish)`delete_on_finish` Whether to delete input files from the disk once they are fully consumed. Requires version 4.11.0 or later. **Type**: `bool` **Default**: `false` ### [](#delimiter)`delimiter` The delimiter to use for splitting values in each record. It must be a single character. **Type**: `string` **Default**: `,` ### [](#lazy_quotes)`lazy_quotes` If set to `true`, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. Requires version 4.1.0 or later. **Type**: `bool` **Default**: `false` ### [](#parse_header_row)`parse_header_row` Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, each message will consist of an array of values from the corresponding CSV row. **Type**: `bool` **Default**: `true` ### [](#paths)`paths[]` A list of file paths to read from. Each file will be read sequentially until the list is exhausted, at which point the input will close. Glob patterns are supported, including super globs (double star). **Type**: `array` ```yaml # Examples: paths: - /tmp/foo.csv - /tmp/bar/*.csv - /tmp/data/**/*.csv ``` --- # Page 44: discord **URL**: https://docs.redpanda.com/connect/components/inputs/discord.md --- # discord --- title: discord latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/discord page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/discord.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/discord.adoc description: Consumes messages posted in a Discord channel. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes messages posted in a Discord channel. This input works by authenticating as a bot using token based authentication. The ID of the newest message consumed and acked is stored in a cache in order to perform a backfill of unread messages each time the input is initialised. Ideally this cache should be persisted across restarts. #### Common ```yml input: label: "" discord: channel_id: "" # No default (required) bot_token: "" # No default (required) cache: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" discord: channel_id: "" # No default (required) bot_token: "" # No default (required) cache: "" # No default (required) cache_key: last_message_id auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` A bot token used for authentication. **Type**: `string` ### [](#cache)`cache` A cache resource to use for performing unread message backfills, the ID of the last message received will be stored in this cache and used for subsequent requests. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier used when storing the ID of the last message received. **Type**: `string` **Default**: `last_message_id` ### [](#channel_id)`channel_id` A discord channel ID to consume messages from. **Type**: `string` --- # Page 45: dynamic **URL**: https://docs.redpanda.com/connect/components/inputs/dynamic.md --- # dynamic --- title: dynamic latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/dynamic page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/dynamic.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/dynamic.adoc description: A special broker type where the inputs are identified by unique labels and can be created, changed and removed during runtime via a REST HTTP interface. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A special broker type where the inputs are identified by unique labels and can be created, changed and removed during runtime via a REST HTTP interface. #### Common ```yml input: label: "" dynamic: inputs: {} prefix: "" ``` #### Advanced ```yml input: label: "" dynamic: inputs: {} prefix: "" bind_http: false ``` ## [](#fields)Fields ### [](#bind_http)`bind_http` Whether to register the REST HTTP endpoints (`/inputs`, `/inputs/{id}`, and so on) for creating, updating and removing inputs at runtime. These endpoints are served on the service-wide HTTP server and accept input configuration over HTTP, so they are disabled by default. Enable them only when that server is secured appropriately, for example bound to a trusted interface or protected with `http.basic_auth`. Requires version 4.111.0 or later. **Type**: `bool` **Default**: `false` ### [](#inputs)`inputs` A map of inputs to statically create. **Type**: `object` **Default**: `{}` ### [](#prefix)`prefix` A path prefix for HTTP endpoints that are registered. **Type**: `string` **Default**: `""` ## [](#endpoints)Endpoints ### [](#get-inputs)GET `/inputs` Returns a JSON object detailing all dynamic inputs, providing information such as their current uptime and configuration. ### [](#get-inputsid)GET `/inputs/{id}` Returns the configuration of an input. ### [](#post-inputsid)POST `/inputs/{id}` Creates or updates an input with a configuration provided in the request body (in YAML or JSON format). ### [](#delete-inputsid)DELETE `/inputs/{id}` Stops and removes an input. ### [](#get-inputsiduptime)GET `/inputs/{id}/uptime` Returns the uptime of an input as a duration string (of the form "72h3m0.5s"), or "stopped" in the case where the input has gracefully terminated. --- # Page 46: file **URL**: https://docs.redpanda.com/connect/components/inputs/file.md --- # file --- title: file latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/file page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/file.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/file.adoc description: Consumes data from files on disk, emitting messages according to a chosen codec. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes data from files on disk, emitting messages according to a chosen codec. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `path` - `mod_time_unix` - `mod_time` (RFC3339) You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" file: paths: [] # No default (required) scanner: lines: {} auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" file: paths: [] # No default (required) scanner: lines: {} delete_on_finish: false auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#delete_on_finish)`delete_on_finish` Whether to delete input files from the disk once they are fully consumed. **Type**: `bool` **Default**: `false` ### [](#paths)`paths[]` A list of paths to consume sequentially. Glob patterns are supported, including super globs (double star). **Type**: `array` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ## [](#examples)Examples ### [](#read-a-bunch-of-csvs)Read a Bunch of CSVs If we wished to consume a directory of CSV files as structured documents we can use a glob pattern and the `csv` scanner: ```yaml input: file: paths: [ ./data/*.csv ] scanner: csv: {} ``` --- # Page 47: gcp_bigquery_select **URL**: https://docs.redpanda.com/connect/components/inputs/gcp_bigquery_select.md --- # gcp_bigquery_select --- title: gcp_bigquery_select latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/gcp_bigquery_select page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_bigquery_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_bigquery_select.adoc description: Executes a SELECT query against BigQuery and creates a message for each row received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a `SELECT` query against BigQuery and creates a message for each row received. Introduced in version 3.63.0. Once the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to execute). ```yml input: label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) auto_replay_nacks: true job_labels: {} priority: "" args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` ## [](#examples)Examples ### [](#word-counts)Word counts Here we query the public corpus of Shakespeare’s works to generate a stream of the top 10 words that are 3 or more characters long: ```yaml input: gcp_bigquery_select: project: sample-project table: bigquery-public-data.samples.shakespeare columns: - word - sum(word_count) as total_count where: length(word) >= ? suffix: | GROUP BY word ORDER BY total_count DESC LIMIT 10 args_mapping: | root = [ 3 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that evaluates to an array of values for the `?` placeholders in the `where` field, in order. Each value is sent to BigQuery as a positional query parameter. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ### [](#credentials_json)`credentials_json` The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#job_labels)`job_labels` A map of labels to add to the query job. **Type**: `object` **Default**: `{}` ### [](#prefix)`prefix` Optional GoogleSQL text to add before the `SELECT` keyword of the generated query, for example a `WITH` clause. **Type**: `string` ### [](#priority)`priority` The priority with which to schedule the query. Requires version 4.7.0 or later. **Type**: `string` **Default**: `""` ### [](#project)`project` The GCP project where the query job runs. **Type**: `string` ### [](#suffix)`suffix` Optional GoogleSQL text to append after the generated query, for example `GROUP BY`, `ORDER BY`, or `LIMIT` clauses. **Type**: `string` ### [](#table)`table` The fully-qualified name of the BigQuery table to query. **Type**: `string` ```yaml # Examples: table: bigquery-public-data.samples.shakespeare ``` ### [](#where)`where` An optional `WHERE` clause to add to the query. The `args_mapping` field populates the placeholder arguments. Placeholders must always be question marks (`?`). **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 48: gcp_cloud_storage **URL**: https://docs.redpanda.com/connect/components/inputs/gcp_cloud_storage.md --- # gcp_cloud_storage --- title: gcp_cloud_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/gcp_cloud_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_cloud_storage.adoc description: Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix. Introduced in version 3.43.0. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `gcs_key` - `gcs_bucket` - `gcs_last_modified` - `gcs_last_modified_unix` - `gcs_content_type` - `gcs_content_encoding` - All user defined metadata You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in [Google Cloud Platform](https://docs.redpanda.com/connect/guides/cloud/gcp/). ### Common ```yml input: label: "" gcp_cloud_storage: bucket: "" # No default (required) prefix: "" credentials_json: "" scanner: to_the_end: {} ``` ### Advanced ```yml input: label: "" gcp_cloud_storage: bucket: "" # No default (required) prefix: "" credentials_json: "" scanner: to_the_end: {} delete_objects: false ``` ## [](#fields)Fields ### [](#bucket)`bucket` The name of the bucket from which to download objects. **Type**: `string` ### [](#credentials_json)`credentials_json` The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#delete_objects)`delete_objects` Whether to remove objects from the Cloud Storage bucket once the messages read from them are delivered without error. An object is kept if delivery fails. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` Optional path prefix, if set only objects with the prefix are consumed. **Type**: `string` **Default**: `""` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 49: gcp_pubsub **URL**: https://docs.redpanda.com/connect/components/inputs/gcp_pubsub.md --- # gcp_pubsub --- title: gcp_pubsub latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/gcp_pubsub page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_pubsub.adoc description: Consumes messages from a GCP Cloud Pub/Sub subscription. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes messages from a GCP Cloud Pub/Sub subscription. For information on how to set up credentials see [this guide](https://cloud.google.com/docs/authentication/production). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `gcp_pubsub_publish_time_unix` - The time at which the message was published to the topic. - `gcp_pubsub_delivery_attempt` - When dead lettering is enabled, this is set to the number of times PubSub has attempted to deliver a message. - `gcp_pubsub_message_id` - The unique identifier of the message. - `gcp_pubsub_ordering_key` - The ordering key of the message. - All message attributes You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" subscription: "" # No default (required) endpoint: "" sync: false max_outstanding_messages: 1000 max_outstanding_bytes: 1000000000 ``` ### Advanced ```yml input: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" subscription: "" # No default (required) endpoint: "" sync: false max_outstanding_messages: 1000 max_outstanding_bytes: 1000000000 create_subscription: enabled: false topic: "" ``` ## [](#fields)Fields ### [](#create_subscription)`create_subscription` Allows you to configure the input subscription and creates if it doesn’t exist. Requires version 4.11.0 or later. **Type**: `object` ### [](#create_subscription-enabled)`create_subscription.enabled` Whether to configure subscription or not. **Type**: `bool` **Default**: `false` ### [](#create_subscription-topic)`create_subscription.topic` Defines the topic that the subscription should be vinculated to. **Type**: `string` **Default**: `""` ### [](#credentials_json)`credentials_json` The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#endpoint)`endpoint` An optional endpoint that overrides the default of `pubsub.googleapis.com:443`. Use this field to connect to a region-specific Pub/Sub endpoint. For a list of valid values, see [Pub/Sub regional endpoints](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_regional_endpoints). Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: endpoint: us-central1-pubsub.googleapis.com:443 # --- endpoint: us-west3-pubsub.googleapis.com:443 ``` ### [](#max_outstanding_bytes)`max_outstanding_bytes` The maximum number of outstanding pending messages to be consumed measured in bytes. **Type**: `int` **Default**: `1000000000` ### [](#max_outstanding_messages)`max_outstanding_messages` The maximum number of outstanding pending messages to be consumed at a given time. **Type**: `int` **Default**: `1000` ### [](#project)`project` The project ID of the target subscription. **Type**: `string` ### [](#subscription)`subscription` The target subscription ID. **Type**: `string` ### [](#sync)`sync` Enable synchronous pull mode. **Type**: `bool` **Default**: `false` --- # Page 50: gcp_spanner_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/gcp_spanner_cdc.md --- # gcp_spanner_cdc --- title: gcp_spanner_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/gcp_spanner_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_spanner_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_spanner_cdc.adoc description: Creates an input that consumes from a spanner change stream. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Consumes change records from a Google Cloud Spanner change stream. Introduced in version 4.56.0. Consumes change records from a Google Cloud Spanner change stream. This input allows you to track and process database changes in real time, making it useful for data replication, event-driven architectures, and maintaining derived data stores. The input reads from a specified change stream within a Spanner database and converts each change record into a message. The message payload contains the change records in JSON format, and metadata is added with details about the Spanner instance, database, and stream. Change streams provide a way to track mutations to your Spanner database tables. For more information about Spanner change streams, refer to the [Google Cloud documentation](https://cloud.google.com/spanner/docs/change-streams). #### Common ```yaml input: label: "" gcp_spanner_cdc: credentials_json: "" project_id: "" # No default (required) instance_id: "" # No default (required) database_id: "" # No default (required) stream_id: "" # No default (required) start_timestamp: "" end_timestamp: "" batching: count: 0 byte_size: 0 period: "" check: "" checkpoint_limit: 1024 auto_replay_nacks: true ``` #### Advanced ```yaml input: label: "" gcp_spanner_cdc: credentials_json: "" project_id: "" # No default (required) instance_id: "" # No default (required) database_id: "" # No default (required) stream_id: "" # No default (required) start_timestamp: "" end_timestamp: "" heartbeat_interval: 10s metadata_table: "" min_watermark_cache_ttl: 5s allowed_mod_types: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) checkpoint_limit: 1024 auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#allowed_mod_types)`allowed_mod_types[]` List of modification types to process. If not specified, all modification types are processed. Allowed values: INSERT, UPDATE, DELETE **Type**: `array` ```yaml # Examples: allowed_mod_types: - INSERT - UPDATE - DELETE ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed at a given time per partition. Increasing this limit enables parallel processing and batching at the output level. Any given partition watermark will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. Requires version 4.107.0 or later. **Type**: `int` **Default**: `1024` ### [](#credentials_json)`credentials_json` Base64-encoded JSON credentials file for authenticating to GCP with a service account. If not provided, Application Default Credentials (ADC) is used. For more information about how to create a service account and obtain the credentials JSON, see the [Google Cloud documentation](https://cloud.google.com/docs/authentication/getting-started). **Type**: `string` **Default**: `""` ### [](#database_id)`database_id` The ID of the Spanner database to read from. This is the name of the database as it appears in the Spanner console or API. For more information about how to create a Spanner database, see the [Google Cloud documentation](https://cloud.google.com/spanner/docs/create-manage-databases). **Type**: `string` ### [](#end_timestamp)`end_timestamp` The timestamp at which to stop reading change records from the change stream. This optional field lets you limit the range of change records processed by the input. The timestamp is exclusive. The timestamp must be in RFC3339 format, such as `2023-10-01T00:00:00Z`. If not set, the input streams change records indefinitely. **Type**: `string` **Default**: `""` ```yaml # Examples: end_timestamp: 2022-01-01T00:00:00Z ``` ### [](#heartbeat_interval)`heartbeat_interval` The interval at which Spanner emits heartbeat records on the change stream when a partition has no data changes. The input uses heartbeat records to advance partition watermarks. Heartbeat records are not emitted as messages. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. For example, `1s` requests a heartbeat record every second. **Type**: `string` **Default**: `10s` ### [](#instance_id)`instance_id` The ID of the Spanner instance to read from. This is the name of the instance as it appears in the Spanner console or API. For more information about how to create a Spanner instance, see the [Google Cloud documentation](https://cloud.google.com/spanner/docs/create-manage-instances). **Type**: `string` ### [](#metadata_table)`metadata_table` The table to store metadata in (default: `cdc_metadata_`). **Type**: `string` **Default**: `""` ### [](#min_watermark_cache_ttl)`min_watermark_cache_ttl` How frequently to query Spanner for the minimum watermark, as a duration string. **Type**: `string` **Default**: `5s` ### [](#project_id)`project_id` The ID of the GCP project that contains the Spanner instance and database. This is the name of the project as it appears in the GCP console or API. For more information about how to create a GCP project, see the [Google Cloud documentation](https://cloud.google.com/resource-manager/docs/creating-managing-projects). **Type**: `string` ### [](#start_timestamp)`start_timestamp` The timestamp at which to start reading change records from the change stream. This optional field lets you limit the range of change records processed by the input. The timestamp is inclusive. The timestamp must be in RFC3339 format, such as `2023-10-01T00:00:00Z`. If not set, the input starts reading from the current time. **Type**: `string` **Default**: `""` ```yaml # Examples: start_timestamp: 2022-01-01T00:00:00Z ``` ### [](#stream_id)`stream_id` The name of the change stream to track. The stream must exist in the Spanner database. To create a change stream, follow the [Google Cloud documentation](https://cloud.google.com/spanner/docs/change-streams/manage). **Type**: `string` --- # Page 51: generate **URL**: https://docs.redpanda.com/connect/components/inputs/generate.md --- # generate --- title: generate latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/generate page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/generate.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/generate.adoc description: Generates messages at a given interval using a Bloblang mapping executed without a context. This allows you to generate messages for testing your pipeline configs. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Generates messages at a given interval using a [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) mapping executed without a context. This allows you to generate messages for testing your pipeline configs. Introduced in version 3.40.0. #### Common ```yml input: label: "" generate: mapping: "" # No default (required) interval: 1s count: 0 batch_size: 1 auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" generate: mapping: "" # No default (required) interval: 1s count: 0 batch_size: 1 auto_replay_nacks: true ``` ## [](#examples)Examples ### [](#cron-scheduled-processing)Cron Scheduled Processing A common use case for the generate input is to trigger processors on a schedule so that the processors themselves can behave similarly to an input. The following configuration reads rows from a PostgreSQL table every 5 minutes. ```yaml input: generate: interval: '@every 5m' mapping: 'root = {}' processors: - sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: foo columns: [ "*" ] ``` ### [](#generate-100-rows)Generate 100 Rows The generate input can be used as a convenient way to generate test data. The following example generates 100 rows of structured data by setting an explicit count. The interval field is set to empty, which means data is generated as fast as the downstream components can consume it. ```yaml input: generate: count: 100 interval: "" mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "is yummy" } } else { { "type": "bar", "bar": "is gross" } } ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batch_size)`batch_size` The number of generated messages that should be accumulated into each batch flushed at the specified interval. Requires version 4.5.0 or later. **Type**: `int` **Default**: `1` ### [](#count)`count` An optional number of messages to generate, if set above 0 the specified number of messages is generated and then the input will shut down. **Type**: `int` **Default**: `0` ### [](#interval)`interval` The time interval at which messages should be generated, expressed either as a duration string or as a cron expression. If set to an empty string messages will be generated as fast as downstream services can process them. Cron expressions can specify a timezone by prefixing the expression with `TZ=`, where the location name corresponds to a file within the IANA Time Zone database. **Type**: `string` **Default**: `1s` ```yaml # Examples: interval: 5s # --- interval: 1m # --- interval: 1h # --- interval: "@every 1s" # --- interval: 0,30 */2 * * * * # --- interval: TZ=Europe/London 30 3-6,20-23 * * * ``` ### [](#mapping)`mapping` A [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) mapping to use for generating messages. **Type**: `string` ```yaml # Examples: mapping: root = "hello world" # --- mapping: root = {"test":"message","id":uuid_v4()} ``` --- # Page 52: git **URL**: https://docs.redpanda.com/connect/components/inputs/git.md --- # git --- title: git latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/git page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/git.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/git.adoc description: A Git input that clones (or pulls) a repository and reads the repository contents. page-git-created-date: "2025-05-02" page-git-modified-date: "2026-10-10" --- Clones a Git repository, reads its contents, then polls for new commits at a configurable interval. Any updates are emitted as new messages. Introduced in version 4.51.0. ```yml input: label: "" git: repository_url: "" # No default (required) branch: main poll_interval: 10s include_patterns: [] exclude_patterns: [] max_file_size: 10485760 checkpoint_cache: "" # No default (optional) checkpoint_key: git_last_commit auth: basic: username: "" password: "" ssh_key: private_key_path: "" private_key: "" passphrase: "" token: value: "" auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `git_file_path` - `git_file_size` - `git_file_mode` - `git_file_modified` - `git_commit` - `git_mime_type` - `git_is_binary` - `git_encoding` (present if the file was base64 encoded) - `git_deleted` (only present if the file was deleted) You can access these metadata fields using function interpolation. ## [](#fields)Fields ### [](#auth)`auth` Options for authenticating with your Git repository. **Type**: `object` ### [](#auth-basic)`auth.basic` Allows you to specify basic authentication. **Type**: `object` ### [](#auth-basic-password)`auth.basic.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auth-basic-username)`auth.basic.username` The username to use for authentication. **Type**: `string` **Default**: `""` ### [](#auth-ssh_key)`auth.ssh_key` Allows you to specify SSH key authentication. **Type**: `object` ### [](#auth-ssh_key-passphrase)`auth.ssh_key.passphrase` The passphrase for your SSH private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auth-ssh_key-private_key)`auth.ssh_key.private_key` Your private SSH key content. When using encrypted keys, you must also set a value for `passphrase`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auth-ssh_key-private_key_path)`auth.ssh_key.private_key_path` The path to your private SSH key file. When using encrypted keys, you must also set a value for `passphrase`. **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Allows you to specify token-based authentication. **Type**: `object` ### [](#auth-token-value)`auth.token.value` The token value to use for token-based authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#branch)`branch` The repository branch to check out. **Type**: `string` **Default**: `main` ### [](#checkpoint_cache)`checkpoint_cache` Specify a [`cache`](https://docs.redpanda.com/connect/components/caches/about/) resource to store the last processed commit hash. After a restart, Redpanda Connect can then continue processing changes from where it left off, avoiding the need to reprocess all detected updates. **Type**: `string` ### [](#checkpoint_key)`checkpoint_key` The key to use when storing the last processed commit hash in the cache. **Type**: `string` **Default**: `git_last_commit` ### [](#exclude_patterns)`exclude_patterns[]` A list of file patterns to exclude. For example, you could choose not to read content from certain Git directories or image files: `'.git/**', '**/*.png'`. These patterns take precedence over `include_patterns`. The following patterns are supported: - Glob patterns: `*`, `/**/`, `?` - Character ranges: `[a-z]`. Escape any character with a special meaning using a backslash. **Type**: `array` **Default**: `[]` ### [](#include_patterns)`include_patterns[]` A list of file patterns to read from. For example, you could read content from only Markdown and YAML files: `'**/*.md', 'configs/*.yaml'`. The following patterns are supported: - Glob patterns: `*`, `/**/`, `?` - Character ranges: `[a-z]`. Escape any character with a special meaning using a backslash. If this field is left empty, all files are read. **Type**: `array` **Default**: `[]` ### [](#max_file_size)`max_file_size` The maximum size of files to read from (in bytes). Files that exceed this limit are skipped. Set to `0` for unlimited file sizes. **Type**: `int` **Default**: `10485760` ### [](#poll_interval)`poll_interval` How frequently this input polls the Git repository for changes, as a duration between polling attempts. **Type**: `string` **Default**: `10s` ```yaml # Examples: poll_interval: 10s ``` ### [](#repository_url)`repository_url` The URL of the Git repository to clone. **Type**: `string` ```yaml # Examples: repository_url: https://github.com/username/repo.git ``` --- # Page 53: hdfs **URL**: https://docs.redpanda.com/connect/components/inputs/hdfs.md --- # hdfs --- title: hdfs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/hdfs page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/hdfs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/hdfs.adoc description: Reads files from a HDFS directory, where each discrete file will be consumed as a single message payload. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads files from a HDFS directory, where each discrete file will be consumed as a single message payload. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `hdfs_name` - `hdfs_path` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ```yml input: label: "" hdfs: hosts: [] # No default (required) user: "" directory: "" # No default (required) ``` ## [](#fields)Fields ### [](#directory)`directory` The directory to consume from. **Type**: `string` ### [](#hosts)`hosts[]` A list of target host addresses to connect to. **Type**: `array` ```yaml # Examples: hosts: localhost:9000 ``` ### [](#user)`user` A user ID to connect as. **Type**: `string` **Default**: `""` --- # Page 54: http_client **URL**: https://docs.redpanda.com/connect/components/inputs/http_client.md --- # http_client --- title: http_client latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/http_client page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/http_client.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/http_client.adoc description: Connects to a server and continuously performs requests for a single message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Connects to a server and continuously requests single messages. #### Common ```yml input: label: "" http_client: url: "" # No default (required) verb: GET headers: {} rate_limit: "" # No default (optional) timeout: 5s payload: "" # No default (optional) stream: enabled: false reconnect: true scanner: lines: {} auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" http_client: url: "" # No default (required) verb: GET headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false payload: "" # No default (optional) drop_empty_bodies: true stream: enabled: false reconnect: true scanner: lines: {} auto_replay_nacks: true ``` ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). You can also add [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to the [`url`](#url) and [`headers`](#headers) fields to implement basic pagination, such as page numbers or tokens, where subsequent requests need to include data from previously-consumed responses. Example: ```yaml input: http_client: url: >- https://api.example.com/search?query=allmyfoos&start_time=${! ( (timestamp_unix()-300).ts_format("2006-01-02T15:04:05Z","UTC").escape_url_query() ) }${! ("&next_token="+this.meta.next_token.not_null()) | "" } verb: GET rate_limit: schedule_searches oauth2: enabled: true token_url: https://api.example.com/oauth2/token client_key: "${EXAMPLE_KEY}" client_secret: "${EXAMPLE_SECRET}" rate_limit_resources: - label: schedule_searches local: count: 1 interval: 30s ``` > 💡 **TIP** > > If pagination requires more complex logic, consider using the [`http` processor](https://docs.redpanda.com/connect/components/processors/http/) combined with a [`generate` input](https://docs.redpanda.com/connect/components/inputs/generate/), which allows you to schedule the processor. ## [](#streaming-messages)Streaming messages If you [enable streaming](#stream-enabled), Redpanda Connect consumes the body of the server response as a continuous stream of data, and breaks the stream down into smaller, logical messages using the [specified scanner](#stream-scanner). This functionality allows you to consume APIs that provide long-lived streamed data feeds, such as stock market feeds. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure and trigger retries with an increasing backoff period between attempts. **Type**: `array` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. Requires version 4.47.0 or later. **Type**: `bool` **Default**: `false` ### [](#drop_empty_bodies)`drop_empty_bodies` Whether to drop empty payloads received from the target server. **Type**: `bool` **Default**: `true` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure where retries should not be attempted. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `array` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, `""` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether to transparently follow redirects, that is, responses with HTTP status codes in the 300-399 range. If set to `false`, the response message includes the body, status, and headers from the redirect response, and the component does not make a request to the URL specified in the `Location` header. Requires version 4.39.0 or later. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys should be added to the HTTP request as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. Requires version 4.21.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). Requires version 3.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#payload)`payload` A payload to deliver for each request (optional). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#proxy_url)`proxy_url` An HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](https://docs.redpanda.com/connect/components/rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The period to wait between failed requests before retrying. For status codes listed in `backoff_on`, this is the starting period, which increases with each attempt up to `max_retry_backoff`. **Type**: `string` **Default**: `1s` ### [](#stream)`stream` Configure streaming mode, where the HTTP connection remains open and messages are processed line-by-line. Set `stream.enabled` to `true` to turn it on. **Type**: `object` ### [](#stream-enabled)`stream.enabled` Enables streaming mode. **Type**: `bool` **Default**: `false` ### [](#stream-reconnect)`stream.reconnect` Whether to automatically reestablish the HTTP connection if it is lost. **Type**: `bool` **Default**: `true` ### [](#stream-scanner)`stream.scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in the `backoff_on` or `drop_on` fields, regardless of this field. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `GET` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` ## [](#examples)Examples Interpolation functions within the `url` and `headers` fields can be used to reference the previously consumed message, which allows simple pagination. ```yaml input: http_client: url: >- https://api.example.com/search?query=allmyfoos&start_time=${! ( (timestamp_unix()-300).ts_format("2006-01-02T15:04:05Z","UTC").escape_url_query() ) }${! ("&next_token="+this.meta.next_token.not_null()) | "" } verb: GET rate_limit: foo_searches oauth2: enabled: true token_url: https://api.example.com/oauth2/token client_key: "${EXAMPLE_KEY}" client_secret: "${EXAMPLE_SECRET}" rate_limit_resources: - label: foo_searches local: count: 1 interval: 30s ``` --- # Page 55: http_server **URL**: https://docs.redpanda.com/connect/components/inputs/http_server.md --- # http_server --- title: http_server latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/http_server page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/http_server.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/http_server.adoc description: Receive messages POSTed over HTTP(S). HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Receive messages sent over HTTP using POST requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. #### Common ```yml input: label: "" http_server: address: "" path: /post ws_path: /post/ws allowed_verbs: - "POST" timeout: 5s rate_limit: "" ``` #### Advanced ```yml input: label: "" http_server: address: "" path: /post ws_path: /post/ws ws_welcome_message: "" ws_rate_limit_message: "" allowed_verbs: - "POST" timeout: 5s rate_limit: "" cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] sync_response: status: "200" headers: Content-Type: "application/octet-stream" metadata_headers: include_prefixes: [] include_patterns: [] tcp: reuse_addr: false reuse_port: false ``` If the `address` config field is left blank, the [service-wide HTTP server](https://docs.redpanda.com/connect/components/http/about/) is used. The field `rate_limit` allows you to specify an optional [`rate_limit` resource](https://docs.redpanda.com/connect/components/rate_limits/about/), which will be applied to each HTTP request made and each websocket payload received. When the rate limit is breached HTTP requests will have a 429 response returned with a Retry-After header. Websocket payloads will be dropped and an optional response payload will be sent as per `ws_rate_limit_message`. ## [](#responses)Responses It’s possible to return a response for each message received using [synchronous responses](https://docs.redpanda.com/connect/guides/sync_responses/). When doing so you can customize headers with the `sync_response` field `headers`, which can also use [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) in the value based on the response message contents. ## [](#endpoints)Endpoints The following fields specify endpoints that are registered for sending messages, and support path parameters of the form `/{foo}`, which are added to ingested messages as metadata. A path ending in `/` will match against all extensions of that path: ### [](#path-defaults-to-post)`path` (defaults to `/post`) This endpoint expects POST requests where the entire request body is consumed as a single message. If the request contains a multipart `content-type` header as per [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html) then the multiple parts are consumed as a batch of messages, where each body part is a message of the batch. ### [](#ws_path-defaults-to-postws)`ws_path` (defaults to `/post/ws`) Creates a websocket connection, where payloads received on the socket are passed through the pipeline as a batch of one message. > ⚠️ **CAUTION: Endpoint caveats** > > Endpoint caveats > > Components within a Redpanda Connect config will register their respective endpoints in a non-deterministic order. This means that establishing precedence of endpoints that are registered via multiple `http_server` inputs or outputs (either within brokers or from cohabiting streams) is not possible in a predictable way. > > This ambiguity makes it difficult to ensure that paths which are both a subset of a path registered by a separate component, and end in a slash (`/`) and will therefore match against all extensions of that path, do not prevent the more specific path from matching against requests. > > It is therefore recommended that you ensure paths of separate components do not collide unless they are explicitly non-competing. > > For example, if you were to deploy two separate `http_server` inputs, one with a path `/foo/` and the other with a path `/foo/bar`, it would not be possible to ensure that the path `/foo/` does not swallow requests made to `/foo/bar`. You may specify an optional `ws_welcome_message`, which is a static payload to be sent to all clients once a websocket connection is first established. It’s also possible to specify a `ws_rate_limit_message`, which is a static payload to be sent to clients that have triggered the servers rate limit. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `http_server_user_agent` - `http_server_request_path` - `http_server_verb` - `http_server_remote_ip` - All headers (only first values are taken) - All query parameters - All path parameters - All cookies If HTTPS is enabled, the following fields are added as well: - `http_server_tls_version` - `http_server_tls_subject` - `http_server_tls_cipher_suite` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#path-switching)Path Switching This example shows an `http_server` input that captures all requests and processes them by switching on that path: ```yaml input: http_server: path: / allowed_verbs: [ GET, POST ] sync_response: headers: Content-Type: application/json processors: - switch: - check: '@http_server_request_path == "/foo"' processors: - mapping: | root.title = "You Got Fooed!" root.result = content().string().uppercase() - check: '@http_server_request_path == "/bar"' processors: - mapping: 'root.title = "Bar Is Slow"' - sleep: # Simulate a slow endpoint duration: 1s ``` ### [](#mock-oauth-2-0-server)Mock OAuth 2.0 Server This example shows an `http_server` input that mocks an OAuth 2.0 Client Credentials flow server at the endpoint `/oauth2_test`: ```yaml input: http_server: path: /oauth2_test allowed_verbs: [ GET, POST ] sync_response: headers: Content-Type: application/json processors: - log: message: "Received request" level: INFO fields_mapping: | root = @ root.body = content().string() - mapping: | root.access_token = "MTQ0NjJkZmQ5OTM2NDE1ZTZjNGZmZjI3" root.token_type = "Bearer" root.expires_in = 3600 - sync_response: {} - mapping: 'root = deleted()' ``` ## [](#fields)Fields ### [](#address)`address` An alternative address to host from. If left empty the service wide address is used. **Type**: `string` **Default**: `""` ### [](#allowed_verbs)`allowed_verbs[]` An array of verbs that are allowed for the `path` endpoint. Requires version 3.33.0 or later. **Type**: `array` **Default**: ```yaml - "POST" ``` ### [](#cert_file)`cert_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#cors)`cors` Adds Cross-Origin Resource Sharing headers. Only valid with a custom `address`. Requires version 3.63.0 or later. **Type**: `object` ### [](#cors-allowed_origins)`cors.allowed_origins[]` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#key_file)`key_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#path)`path` The endpoint path to listen for POST requests. **Type**: `string` **Default**: `/post` ### [](#rate_limit)`rate_limit` An optional [rate limit](https://docs.redpanda.com/connect/components/rate_limits/about/) to throttle requests by. **Type**: `string` **Default**: `""` ### [](#sync_response)`sync_response` Customize messages returned via [synchronous responses](https://docs.redpanda.com/connect/guides/sync_responses/). **Type**: `object` ### [](#sync_response-headers)`sync_response.headers` Specify headers to return with synchronous responses. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: ```yaml Content-Type: "application/octet-stream" ``` ### [](#sync_response-metadata_headers)`sync_response.metadata_headers` Specify criteria for which metadata values are added to the response as headers. **Type**: `object` ### [](#sync_response-metadata_headers-include_patterns)`sync_response.metadata_headers.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#sync_response-metadata_headers-include_prefixes)`sync_response.metadata_headers.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#sync_response-status)`sync_response.status` Specify the status code to return with synchronous responses. This is a string value, which allows you to customize it based on resulting payloads and their metadata. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `200` ```yaml # Examples: status: ${! json("status") } # --- status: ${! meta("status") } ``` ### [](#tcp)`tcp` TCP listener configuration for the HTTP server. Only valid with a custom `address`. Requires version 4.78.0 or later. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#timeout)`timeout` Timeout for requests. If a consumed messages takes longer than this to be delivered the connection is closed, but the message may still be delivered. **Type**: `string` **Default**: `5s` ### [](#ws_path)`ws_path` The endpoint path to create websocket connections from. **Type**: `string` **Default**: `/post/ws` ### [](#ws_rate_limit_message)`ws_rate_limit_message` An optional message to delivery to websocket connections that are rate limited. **Type**: `string` **Default**: `""` ### [](#ws_welcome_message)`ws_welcome_message` An optional message to deliver to fresh websocket connections. **Type**: `string` **Default**: `""` --- # Page 56: inproc **URL**: https://docs.redpanda.com/connect/components/inputs/inproc.md --- # inproc --- title: inproc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/inproc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/inproc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/inproc.adoc description: Directly connects to an output within the same Redpanda Connect process by a chosen ID, to link isolated streams when running in streams mode. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- ```yml input: label: "" inproc: "" ``` Directly connect to an output within a Redpanda Connect process by referencing it by a chosen ID. This allows you to hook up isolated streams whilst running Redpanda Connect in [streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/), it is NOT recommended that you connect the inputs of a stream with an output of the same stream, as feedback loops can lead to deadlocks in your message flow. It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs. --- # Page 57: jira **URL**: https://docs.redpanda.com/connect/components/inputs/jira.md --- # jira --- title: jira latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/jira page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/jira.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/jira.adoc description: Streams Jira issues, comments, or changelog entries via JQL with incremental polling. page-git-created-date: "2026-07-20" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams Jira issues, comments, or changelog entries via JQL with incremental polling. Introduced in version 4.100.0. Periodically queries Jira’s REST API using a JQL filter and emits one message per resource. The cursor (max issue `updated` timestamp, plus the set of issue versions already emitted at the boundary) is persisted via the configured cache resource after every fully-acknowledged page, so progress survives restarts (including mid-backfill), and boundary issues are not re-emitted on every poll. Set `base_url` to the URL of your Jira instance, for example `[https://your-domain.atlassian.net](https://your-domain.atlassian.net)`. Authentication uses API token (email + token) basic auth. The `backoff` settings govern the adaptive backoff applied to 429 responses; retries of 502/503/504 responses use a fixed three-attempt policy. Each message body is the raw JSON of the resource. Metadata fields: - `jira_id` - issue key (issues) / comment ID / changelog history ID - `jira_issue_key` - parent issue key (omitted for resource=issues) - `jira_project` - project key - `jira_updated` - RFC 3339 timestamp of the resource - `jira_event_type` - "issue" / "comment" / "changelog" - `jira_self` - Jira API URL of the resource Limitations (v1): OAuth and the worklogs resource are not yet supported. For resource=comments and resource=changelog, only the first page of child resources (up to ~50 comments or ~100 changelog entries per issue update) is emitted; a WARN is logged when truncation is detected. To fetch the full child set for issues that exceed this limit, query the Jira REST API directly. #### Common ```yml input: label: "" jira: auth: email: "" # No default (required) api_token: "" # No default (required) resource: issues jql: "" fields: - "*all" expand: [] page_size: 50 poll_interval: 60s cursor: cache: "" # No default (required) key: "" overlap: 60s auto_replay_nacks: true base_url: "" # No default (required) timeout: 5s ``` #### Advanced ```yml input: label: "" jira: auth: email: "" # No default (required) api_token: "" # No default (required) resource: issues jql: "" fields: - "*all" expand: [] page_size: 50 poll_interval: 60s cursor: cache: "" # No default (required) key: "" overlap: 60s auto_replay_nacks: true base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#auth)`auth` API token authentication. **Type**: `object` ### [](#auth-api_token)`auth.api_token` Jira API token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#auth-email)`auth.email` Email or username of the Jira account. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` The base URL of the target service (for example, `[https://api.example.com](https://api.example.com)`). TLS is enabled automatically for `https` URLs. **Type**: `string` ### [](#cursor)`cursor` Cursor checkpoint storage. **Type**: `object` ### [](#cursor-cache)`cursor.cache` Name of a cache resource used to persist the cursor. **Type**: `string` ### [](#cursor-key)`cursor.key` Cache key. Defaults to `redpanda_connect_jira_input_`. **Type**: `string` **Default**: `""` ### [](#cursor-overlap)`cursor.overlap` Widens `updated >= cursor - overlap` to absorb minute-boundary effects. Jira JQL’s `updated` operator has minute precision, so this should be set to at least 1m to have an effect. **Type**: `string` **Default**: `60s` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#expand)`expand[]` Jira `expand` query parameter. The input automatically adds `changelog` when resource=changelog. **Type**: `array` **Default**: `[]` ### [](#fields-2)`fields[]` Jira `fields` query parameter - narrow this for throughput. **Type**: `array` **Default**: ```yaml - "*all" ``` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#jql)`jql` Jira JQL filter. The input appends an `updated >= cursor` predicate and `ORDER BY updated ASC, key ASC`. Empty means all issues visible to the principal. **Type**: `string` **Default**: `""` ### [](#page_size)`page_size` Issues per Jira page (Jira max 100). **Type**: `int` **Default**: `50` ### [](#poll_interval)`poll_interval` Time to wait between polls once the input has caught up. Minimum 10s. **Type**: `string` **Default**: `60s` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#resource)`resource` Which Jira resource to emit. **Type**: `string` **Default**: `issues` **Options**: `issues`, `comments`, `changelog` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` --- # Page 58: kafka_franz **URL**: https://docs.redpanda.com/connect/components/inputs/kafka_franz.md --- # kafka_franz --- title: kafka_franz latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/kafka_franz page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/kafka_franz.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/kafka_franz.adoc description: A Kafka input using the Franz Kafka client library. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) components. A Kafka input using the [Franz Kafka client library](https://github.com/twmb/franz-go). Introduced in version 3.61.0. #### Common ```yml input: label: "" kafka_franz: seed_brokers: [] # No default (required) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" kafka_franz: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) checkpoint_limit: 1024 commit_period: 5s multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) topic_lag_refresh_period: 5s auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) ``` When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. This input often out-performs the traditional `kafka` input and provides more useful logs and error messages. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/) that applies to individual topic partitions, so that the input batches messages together before flushing them for processing. Batching can improve performance and is useful for windowed processing. Batching this way preserves the ordering of topic partitions. Requires version 4.15.0 or later. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. Requires version 4.21.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown. Requires version 4.3.0 or later. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.53.0 or later. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics specified in the `topics` field are automatically distributed across consumers that share the consumer group. - Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. This field is equivalent to the Java setting `fetch.max.bytes`. Requires version 4.39.0 or later. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. Requires version 4.39.0 or later. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). This field is equivalent to the Java setting `fetch.max.wait.ms`. Requires version 4.45.0 or later. **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. Requires version 4.39.0 or later. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. This value must be lower than `session_timeout`, and should be no higher than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a `consumer_group`, set `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. The value must be unique per consumer within the group. When you assign an instance ID, the client does not leave the consumer group when it closes. To remove the client from the group, you must use an external admin command on behalf of the instance ID. Requires version 4.46.0 or later. **Type**: `string` **Default**: `""` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#multi_header)`multi_header` Decode headers into lists to allow the handling of multiple values with the same key. Requires version 4.10.0 or later. **Type**: `bool` **Default**: `false` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. Requires version 4.21.0 or later. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a `consumer_group`, `rebalance_timeout` sets how long consumer group members can take to complete their work and commit offsets after a rebalance has begun. The time that a member takes to detect the rebalance (from a heartbeat) counts against this timeout. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. Requires version 4.69.0 or later. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. Requires version 4.69.0 or later. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. Requires version 4.53.0 or later. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. Requires version 4.3.0 or later. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. Requires version 4.53.0 or later. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.62.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between consumer lag refreshes. During each cycle, this input asks the brokers for the consumer group’s committed offsets and the partition end offsets, and records the difference (the number of unread messages) for each topic partition in the consumer lag metric and the `kafka_lag` metadata field. Lag is only refreshed when `consumer_group` is set. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.54.1 or later. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. You can list multiple comma-separated topics in a single element. If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusive. Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. Requires version 4.53.0 or later. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | --- # Page 59: kafka **URL**: https://docs.redpanda.com/connect/components/inputs/kafka.md --- # kafka --- title: kafka latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/kafka page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/kafka.adoc description: Connects to Kafka brokers and consumes one or more topics. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) components. Connects to Kafka brokers and consumes one or more topics. #### Common ```yml input: label: "" kafka: addresses: [] # No default (required) topics: [] # No default (required) target_version: "" # No default (optional) consumer_group: "" checkpoint_limit: 1024 auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" kafka: addresses: [] # No default (required) topics: [] # No default (required) target_version: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" access_token: "" token_cache: "" token_key: "" consumer_group: "" client_id: benthos instance_id: "" # No default (optional) rack_id: "" start_from_oldest: true checkpoint_limit: 1024 auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) commit_period: 1s max_processing_period: 100ms extract_tracing_map: "" # No default (optional) group: session_timeout: 10s heartbeat_interval: 3s rebalance_timeout: 60s fetch_buffer_cap: 256 multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Offsets are managed within Kafka under the specified consumer group, and partitions for each topic are automatically balanced across members of the consumer group. The Kafka input allows parallel processing of messages from different topic partitions, and messages of the same topic partition are processed with a maximum parallelism determined by the field [`checkpoint_limit`](#checkpoint_limit). To enforce ordered processing of partition messages, set the [`checkpoint_limit`](#checkpoint_limit) to `1`, which makes sure that a message is only processed after the previous message is delivered. Batching messages before processing can be enabled using the [`batching`](#batching) field, and this batching is performed per-partition such that messages of a batch will always originate from the same partition. This batching mechanism is capable of creating batches of greater size than the [`checkpoint_limit`](#checkpoint_limit), in which case the next batch will only be created upon delivery of the current one. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All existing message headers (version 0.11+) The field `kafka_lag` is the calculated difference between the high water mark offset of the partition at the time of ingestion and the current message offset. You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#ordering)Ordering By default messages of a topic partition can be processed in parallel, up to a limit determined by the field `checkpoint_limit`. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level. ## [](#troubleshooting)Troubleshooting If you’re seeing issues writing to or reading from Kafka with this component then it’s worth trying out the newer [`kafka_franz` input](https://docs.redpanda.com/connect/components/inputs/kafka_franz/). - I’m seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tls-enabled) if applicable. ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of broker addresses to connect to. List items that contain commas are expanded into multiple addresses. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9092" # --- addresses: - "localhost:9041,localhost:9042" # --- addresses: - "localhost:9041" - "localhost:9042" ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages of the same topic and partition that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual partitions. Any given offset will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. Requires version 3.33.0 or later. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` The client ID sent to brokers with every request, which lets brokers attribute requests to this client in their logs, metrics, and client quotas. **Type**: `string` **Default**: `benthos` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown. **Type**: `string` **Default**: `1s` ### [](#consumer_group)`consumer_group` An identifier for the consumer group of the connection. This field can be explicitly made empty in order to disable stored offsets for the consumed topic partitions. **Type**: `string` **Default**: `""` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 3.45.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#fetch_buffer_cap)`fetch_buffer_cap` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `256` ### [](#group)`group` Tuning parameters for consumer group synchronization. **Type**: `object` ### [](#group-heartbeat_interval)`group.heartbeat_interval` A period in which heartbeats should be sent out. **Type**: `string` **Default**: `3s` ### [](#group-rebalance_timeout)`group.rebalance_timeout` A period after which rebalancing is abandoned if unresolved. **Type**: `string` **Default**: `60s` ### [](#group-session_timeout)`group.session_timeout` A period after which a consumer of the group is kicked after no heartbeats. **Type**: `string` **Default**: `10s` ### [](#instance_id)`instance_id` When you specify a `consumer_group`, assign a unique value to `instance_id` for each input so that brokers can identify it across restarts of this process and avoid unnecessary rebalances. Requires version 4.46.0 or later. **Type**: `string` ### [](#max_processing_period)`max_processing_period` A maximum estimate for the time taken to process a message, this is used for tuning consumer group synchronization. **Type**: `string` **Default**: `100ms` ### [](#multi_header)`multi_header` Decode headers into lists to allow handling of multiple values with the same key Requires version 4.10.0 or later. **Type**: `bool` **Default**: `false` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-access_token)`sasl.access_token` A static OAUTHBEARER access token **Type**: `string` **Default**: `""` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism, if left empty SASL authentication is not used. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS. | | SCRAM-SHA-256 | Authentication using the SCRAM-SHA-256 mechanism. | | SCRAM-SHA-512 | Authentication using the SCRAM-SHA-512 mechanism. | | none | Default, no SASL authentication. | ### [](#sasl-password)`sasl.password` A PLAIN password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-token_cache)`sasl.token_cache` Instead of using a static `access_token` allows you to query a [`cache`](https://docs.redpanda.com/connect/components/caches/about/) resource to fetch OAUTHBEARER tokens from **Type**: `string` **Default**: `""` ### [](#sasl-token_key)`sasl.token_key` Required when using a `token_cache`, the key to query the cache with for tokens. **Type**: `string` **Default**: `""` ### [](#sasl-user)`sasl.user` A PLAIN username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#start_from_oldest)`start_from_oldest` Determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. The setting is applied when creating a new consumer group or the saved offset no longer exists. **Type**: `bool` **Default**: `true` ### [](#target_version)`target_version` The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version. **Type**: `string` ```yaml # Examples: target_version: 2.1.0 # --- target_version: 3.1.0 ``` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.62.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topics)`topics[]` A list of topics to consume from. Multiple comma separated topics can be listed in a single element. Partitions are automatically distributed across consumers of a topic. Alternatively, it’s possible to specify explicit partitions to consume from with a colon after the topic name. For example `foo:0` would consume the partition 0 of the topic foo. This syntax supports ranges. For example `foo:0-10` would consume partitions 0 through to 10 inclusive. Requires version 3.33.0 or later. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` --- # Page 60: microsoft_sql_server_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc.md --- # microsoft_sql_server_cdc --- title: microsoft_sql_server_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/microsoft_sql_server_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/microsoft_sql_server_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/microsoft_sql_server_cdc.adoc description: Enables Change Data Capture by consuming from Microsoft SQL Server's change tables. page-git-created-date: "2025-10-24" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Enables Change Data Capture by consuming from Microsoft SQL Server’s change tables. Introduced in version 4.67.5. #### Common ```yaml input: label: "" microsoft_sql_server_cdc: connection_string: "" # No default (required) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: rpcn.CdcCheckpointCache checkpoint_cache_connection_string: "" # No default (optional) checkpoint_cache_key: microsoft_sql_server_cdc checkpoint_limit: 1024 stream_backoff_interval: 5s auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yaml input: label: "" microsoft_sql_server_cdc: connection_string: "" # No default (required) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: rpcn.CdcCheckpointCache checkpoint_cache_connection_string: "" # No default (optional) checkpoint_cache_key: microsoft_sql_server_cdc checkpoint_limit: 1024 stream_backoff_interval: 5s auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Streams changes from a Microsoft SQL Server database for Change Data Capture (CDC). Additionally, if `stream_snapshot` is set to true, then the existing data in the database is also streamed too. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `database_schema` (The database schema for the table where the message originates from) - `schema` (The table schema in benthos common schema format, compatible with processors like parquet\_encode) - `table` (Name of the table that the message originated from) - `operation` (Type of operation that generated the message: "read", "delete", "insert", or "update\_before" and "update\_after". "read" is from messages that are read in the initial snapshot phase.) - `lsn` (The commit Log Sequence Number of the change, from the change table column `__$start_lsn`. Not present on snapshot (`read`) messages.) - `seqval` (The position of the change in the transaction log, from the change table column `__$seqval`, as a hexadecimal string with a `0x` prefix. Not present on snapshot (`read`) messages.) - `command_id` (The order of the statement within its transaction, from the change table column `__$command_id`. Not present on snapshot (`read`) messages.) ## [](#permissions)Permissions To use the default Microsoft SQL Server cache, the user must have permissions to create tables and stored procedures. Refer to [`checkpoint_cache_table_name`](#checkpoint_cache_table_name) for additional details. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) to store the current Log Sequence Number (LSN) position. The cache stores the highest LSN that has been successfully delivered downstream, which allows Redpanda Connect to resume from the last processed position after a restart, rather than consume the entire state of the change table. If not set, the default Microsoft SQL Server based cache is used. See `checkpoint_cache_table_name` for more information. **Type**: `string` ### [](#checkpoint_cache_connection_string)`checkpoint_cache_connection_string` An optional connection string for a remote Microsoft SQL Server to use for the checkpoint cache. When set, this creates the checkpoint cache table on the remote server instead of the source database. If `checkpoint_cache` is also set, that takes precedence. Requires version 4.87.0 or later. **Type**: `string` ```yaml # Examples: checkpoint_cache_connection_string: sqlserver://username:password@remotehost/instance?param1=value¶m2=value ``` ### [](#checkpoint_cache_key)`checkpoint_cache_key` The key to use to store the snapshot position in `checkpoint_cache`. An alternative key can be provided if multiple CDC inputs share the same cache. Requires version 4.67.0 or later. **Type**: `string` **Default**: `microsoft_sql_server_cdc` ### [](#checkpoint_cache_table_name)`checkpoint_cache_table_name` The multipart identifier for the checkpoint cache table name. If no `checkpoint_cache` field is specified, this input will automatically create a table and stored procedure under the `rpcn` schema to act as a checkpoint cache. This table stores the latest processed Log Sequence Number (LSN) that has been successfully delivered, allowing Redpanda Connect to resume from that point upon restart rather than reconsume the entire change table. Requires version 4.67.0 or later. **Type**: `string` **Default**: `rpcn.CdcCheckpointCache` ```yaml # Examples: checkpoint_cache_table_name: dbo.checkpoint_cache ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed concurrently before applying back pressure. Higher values enable more parallel processing and batching at the output level, but increase memory usage. To preserve at-least-once delivery guarantees, a given Log Sequence Number (LSN) is only acknowledged after all messages under that offset are delivered. **Type**: `int` **Default**: `1024` ### [](#connection_string)`connection_string` The connection string for the Microsoft SQL Server database. Use the format `sqlserver://username:password@host/instance?param1=value¶m2=value`. For Windows Authentication, use `sqlserver://host/instance?trusted_connection=yes`. Include additional parameters like `TrustServerCertificate=true` for self-signed certificates or `encrypt=disable` to disable encryption. **Type**: `string` ```yaml # Examples: connection_string: sqlserver://username:password@host/instance?param1=value¶m2=value ``` ### [](#exclude)`exclude[]` Regular expressions for tables to exclude from CDC streaming. Use this to filter out specific tables from the include patterns. Table names should follow the `schema.table` format. Exclude patterns are applied after include patterns, allowing you to include broad patterns while excluding specific tables. **Type**: `array` ```yaml # Examples: exclude: dbo.privatetable ``` ### [](#include)`include[]` Regular expressions for tables to include in CDC streaming. Specify table names using the format `schema.table` (such as `dbo.orders`, `sales.customers`). Each pattern is treated as a regular expression, allowing wildcards and pattern matching. All specified tables must have CDC enabled in SQL Server. **Type**: `array` ```yaml # Examples: include: dbo.products ``` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` The maximum number of tables to read in parallel during the initial snapshot. Each table is read by its own reader. Requires version 4.69.0 or later. **Type**: `int` **Default**: `1` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of rows to stream in a single batch during the initial snapshot phase. Larger batch sizes can improve throughput for initial data loads but may increase memory usage. This setting only applies when `stream_snapshot` is enabled. **Type**: `int` **Default**: `1000` ### [](#stream_backoff_interval)`stream_backoff_interval` The interval to wait before checking for new changes after a pass over the change tables completes. Each pass drains changes up to the maximum LSN observed as the pass began, then waits for this interval while SQL Server’s capture job continues to publish changes. For low-traffic tables, increasing this value reduces query load on the server. For high-traffic tables, a longer interval directly reduces throughput, because the input sits idle for the full interval between passes. Consider lowering it towards `500ms`, which matches the default poll interval of comparable CDC systems. Use Go duration format such as `500ms`, `5s`, or `1m`. **Type**: `string` **Default**: `5s` ```yaml # Examples: stream_backoff_interval: 500ms # --- stream_backoff_interval: 5s # --- stream_backoff_interval: 1m ``` ### [](#stream_snapshot)`stream_snapshot` Whether to stream a snapshot of all existing data before streaming CDC changes. When set to `true`, the connector first queries all existing table data, then switches to streaming incremental changes from the change tables. When set to `false`, no snapshot is taken and, on first run, streaming begins from the start of each table’s existing change table: every change retained by SQL Server’s CDC capture and cleanup jobs (three days by default) is replayed, not only changes from the current LSN onward. To begin from the present on a table that already holds change history, disable and re-enable CDC on the table immediately before starting the pipeline so that its change table starts empty. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` --- # Page 61: mongodb_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/mongodb_cdc.md --- # mongodb_cdc --- title: mongodb_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/mongodb_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mongodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mongodb_cdc.adoc description: Streams changes from a MongoDB replica set. page-git-created-date: "2025-03-11" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data changes from a MongoDB replica set, using MongoDB’s [change streams](https://www.mongodb.com/docs/manual/changeStreams/) to capture data updates. Introduced in version 4.48.1. #### Common ```yml input: label: "" mongodb_cdc: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collections: [] # No default (required) checkpoint_key: mongodb_cdc_checkpoint checkpoint_cache: "" # No default (required) checkpoint_interval: 5s checkpoint_limit: 1000 read_batch_size: 1000 read_max_wait: 1s stream_snapshot: false snapshot_parallelism: 1 auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" mongodb_cdc: url: "" # No default (required) database: "" # No default (required) username: "" password: "" aws: enabled: false region: "" # No default (optional) session_duration: 1h id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) collections: [] # No default (required) checkpoint_key: mongodb_cdc_checkpoint checkpoint_cache: "" # No default (required) checkpoint_interval: 5s checkpoint_limit: 1000 checkpoint_write_timeout: 10s on_unresumable_position: fail read_batch_size: 1000 read_max_wait: 1s stream_snapshot: false snapshot_parallelism: 1 snapshot_auto_bucket_sharding: false document_mode: update_lookup json_marshal_mode: canonical app_name: benthos auto_replay_nacks: true ``` ## [](#prerequisites)Prerequisites - MongoDB version 6 or later - Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. - A MongoDB database running as a [replica set](https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb) or in a [sharded cluster](https://www.mongodb.com/docs/manual/sharding/) using replica set [protocol version 1](https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion). - A MongoDB database using the [WiredTiger](https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger) storage engine. ## [](#enable-connectivity-from-cloud-based-data-sources)Enable connectivity from cloud-based data sources To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules. ## [](#data-capture-method)Data capture method The `mongodb_cdc` input uses [change streams](https://www.mongodb.com/docs/manual/changeStreams/) to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and [required collections](#collections). For more information, see [`document_mode` options](#document_mode) and the [MongoDB documentation](https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre—​and-post-images). ## [](#data-replication)Data replication Redpanda Connect allows you to specify which [database collections](#collections) in your source database to receive changes from. You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates. - Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded [operations log (oplog)](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) position. - Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections: 1. Records the latest oplog position. 2. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing: 1. If [`snapshot_auto_bucket_sharding`](#snapshot_auto_bucket_sharding) is set to `false`, the internal `$splitVector` command is used to compute shards. 2. If [`snapshot_auto_bucket_sharding`](#snapshot_auto_bucket_sharding) is set to `true`, the [`$bucketAuto`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/) command is used instead. This setting is for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. 3. This input then uses the number of connections specified in [`snapshot-parallelism`](#snapshot_parallelism) to read the selected collections. > 📝 **NOTE** > > If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the [`checkpoint_cache`](#checkpoint_cache). 4. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing. ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the [`checkpoint_cache`](#checkpoint_cache). ## [](#metadata)Metadata Each message emitted by this plugin has the following metadata: - `operation`: either "insert", "replace", "delete" or "update" for changes streamed. Documents from the initial snapshot have the operation set to "read". - `collection`: the collection the document was written to. - `operation_time`: the oplog time for when this operation occurred. - `schema`: the collection schema in benthos common schema format (set as immutable metadata). Extracted from the collection’s `$jsonSchema` validator if available, otherwise inferred from the first document seen. Not present on messages where no schema could be determined (for example, deletes without pre-images when no prior schema is cached). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication using the `MONGODB-AWS` mechanism, for example against MongoDB Atlas. When enabled, IAM credentials are used instead of a static username and password. Role-derived session credentials are resolved when the component connects and are re-resolved whenever it reconnects. The `mongodb` processor and cache establish their client once at creation and cannot refresh expiring session credentials, so `role`, `roles` and session tokens are rejected for those components; use the ambient credential chain or long-lived access keys with them. For long-running pipelines, prefer the ambient credential chain (leave keys and roles unset), which the driver refreshes automatically. Requires version 4.106.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication using the driver-native `MONGODB-AWS` mechanism. The MongoDB Atlas database user must be created with the AWS IAM authentication type, and connections require TLS. When no static credentials or roles are configured, the ambient AWS credential chain (environment variables, EC2 instance profile, EKS pod role) is used and expiring credentials are refreshed automatically. **Type**: `bool` **Default**: `false` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region used when assuming roles (for STS calls). Only used when `role` or `roles` are configured; the ambient and static-key paths ignore it. If no region is specified then the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Cannot be combined with `roles`; use the `roles` array instead when chaining multiple roles. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. Cannot be combined with `role`. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-session_duration)`aws.session_duration` The duration of the STS session requested when assuming roles. AWS requires at least 15 minutes and caps sessions created through role chaining at one hour. Only used when `role` or `roles` are configured. For long-running pipelines, prefer the ambient credential chain over a fixed session duration, since the driver refreshes ambient credentials automatically as they near expiry. When using role assumption with this input, credentials are freshly resolved after the initial snapshot completes, so the streaming phase starts with a full session. The snapshot itself must still complete within a single session duration: snapshot progress is not checkpointed, so a credential expiry mid-snapshot restarts the snapshot from scratch after reconnecting. Once the snapshot completes and is fully acknowledged, its position is checkpointed, so later restarts resume the stream without re-running the snapshot. **Type**: `string` **Default**: `1h` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#checkpoint_cache)`checkpoint_cache` Specify a [`cache` resource](https://docs.redpanda.com/connect/components/caches/about/) to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates. **Type**: `string` ### [](#checkpoint_interval)`checkpoint_interval` The interval between writing checkpoints to the cache. **Type**: `string` **Default**: `5s` ### [](#checkpoint_key)`checkpoint_key` The key identifier used to store the stream’s resume position in `checkpoint_cache`. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key. **Type**: `string` **Default**: `mongodb_cdc_checkpoint` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, the stream’s resume position is only checkpointed once all messages before it are delivered. **Type**: `int` **Default**: `1000` ### [](#checkpoint_write_timeout)`checkpoint_write_timeout` Bounds the checkpoint writes that run outside the normal read loop - storing the position a completed snapshot reached, and clearing a position that can no longer be resumed from - so that a shutdown racing either of them is not extended indefinitely by a slow cache. Raise this for slow remote caches (for example `redis` or `dynamodb`), where losing the post-snapshot write costs a full re-snapshot on the next start. Requires version 4.106.0 or later. **Type**: `string` **Default**: `10s` ### [](#collections)`collections[]` A list of collections to stream changes from. Specify each collection name as a separate item. **Type**: `array` ### [](#database)`database` The name of the MongoDB database to stream changes from. **Type**: `string` ### [](#document_mode)`document_mode` The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes. **Type**: `string` **Default**: `update_lookup` | Option | Summary | | --- | --- | | partial_update | In this mode update operations only have a description of the update operation, which follows the following schema: { "_id": , "operations": [ # type == set means that the value was updated like so: # root.foo."bar.baz" = "world" {"path": ["foo", "bar.baz"], "type": "set", "value":"world"}, # type == unset means that the value was deleted like so: # root.qux = deleted() {"path": ["qux"], "type": "unset", "value": null}, # type == truncatedArray means that the array at that path was truncated to value number of elements # root.array = this.array.slice(2) {"path": ["array"], "type": "truncatedArray", "value": 2} ] } | | pre_and_post_images | Uses pre and post image collection to emit the full documents for update and delete operations. To use and configure this mode see the setup steps in the MongoDB documentation. | | update_lookup | In this mode insert, replace and update operations have the full document emitted and deletes only have the _id field populated. Documents updates lookup the full document. This corresponds to the updateLookup option, see the MongoDB documentation for more information. | ### [](#json_marshal_mode)`json_marshal_mode` Controls the format used to convert a message from BSON to JSON when it is received by Redpanda Connect. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information. | ### [](#on_unresumable_position)`on_unresumable_position` What to do when the stored stream position can no longer be resumed from (for example it has aged out of the oplog) and `stream_snapshot` is disabled, so there is no snapshot to recover with: `fail` stops the input with an error, preserving the checkpoint for inspection; `reset` clears the checkpoint and restarts streaming from the current oplog position, skipping the changes between the lost position and now. When `stream_snapshot` is enabled this field has no effect: recovery re-runs the snapshot, which loses nothing. Requires version 4.106.0 or later. **Type**: `string` **Default**: `fail` **Options**: `fail`, `reset` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#read_batch_size)`read_batch_size` The number of documents to fetch in each message batch from MongoDB. **Type**: `int` **Default**: `1000` ### [](#read_max_wait)`read_max_wait` The maximum duration MongoDB waits to accumulate `read_batch_size` documents on a change stream before returning the batch to Redpanda Connect. **Type**: `string` **Default**: `1s` ### [](#snapshot_auto_bucket_sharding)`snapshot_auto_bucket_sharding` If set to `true`, uses the [`$bucketAuto`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/) aggregation stage instead of the default `splitVector` command to split the snapshot data into chunks for processing. This allows parallel collection reading in environments where privileged access to the MongoDB cluster is not allowed, such as MongoDB Atlas, where the `splitVector` command is not available. To enable parallel processing in these environments: - Set this field to `true`. - Set `stream_snapshot` to `true`. - Increase `snapshot_parallelism` to a value greater than `1`. **Type**: `bool` **Default**: `false` ### [](#snapshot_parallelism)`snapshot_parallelism` Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing. This field is only applicable when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` --- # Page 62: mongodb **URL**: https://docs.redpanda.com/connect/components/inputs/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/mongodb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mongodb.adoc description: Executes a query and creates a message for each document received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a query and creates a message for each document received. Introduced in version 3.64.0. Once the documents from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to execute). #### Common ```yml input: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true batch_size: 0 # No default (optional) sort: {} # No default (optional) limit: 0 # No default (optional) ``` #### Advanced ```yml input: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos aws: enabled: false region: "" # No default (optional) session_duration: 1h id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) collection: "" # No default (required) operation: find json_marshal_mode: canonical query: "" # No default (required) auto_replay_nacks: true batch_size: 0 # No default (optional) sort: {} # No default (optional) limit: 0 # No default (optional) ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. Requires version 4.32.0 or later. **Type**: `string` **Default**: `benthos` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication using the `MONGODB-AWS` mechanism, for example against MongoDB Atlas. When enabled, IAM credentials are used instead of a static username and password. Role-derived session credentials are resolved when the component connects and are re-resolved whenever it reconnects. The `mongodb` processor and cache establish their client once at creation and cannot refresh expiring session credentials, so `role`, `roles` and session tokens are rejected for those components; use the ambient credential chain or long-lived access keys with them. For long-running pipelines, prefer the ambient credential chain (leave keys and roles unset), which the driver refreshes automatically. Requires version 4.106.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication using the driver-native `MONGODB-AWS` mechanism. The MongoDB Atlas database user must be created with the AWS IAM authentication type, and connections require TLS. When no static credentials or roles are configured, the ambient AWS credential chain (environment variables, EC2 instance profile, EKS pod role) is used and expiring credentials are refreshed automatically. **Type**: `bool` **Default**: `false` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region used when assuming roles (for STS calls). Only used when `role` or `roles` are configured; the ambient and static-key paths ignore it. If no region is specified then the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Cannot be combined with `roles`; use the `roles` array instead when chaining multiple roles. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. Cannot be combined with `role`. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-session_duration)`aws.session_duration` The duration of the STS session requested when assuming roles. AWS requires at least 15 minutes and caps sessions created through role chaining at one hour. Only used when `role` or `roles` are configured. For long-running pipelines, prefer the ambient credential chain over a fixed session duration, since the driver refreshes ambient credentials automatically as they near expiry. **Type**: `string` **Default**: `1h` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#batch_size)`batch_size` A explicit number of documents to batch up before flushing them for processing. Must be greater than `0`. Operations: `find`, `aggregate` Requires version 4.26.0 or later. **Type**: `int` ```yaml # Examples: batch_size: 1000 ``` ### [](#collection)`collection` The collection to select from. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#json_marshal_mode)`json_marshal_mode` The json\_marshal\_mode setting is optional and controls the format of the output message. Requires version 4.7.0 or later. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information. | ### [](#limit)`limit` An explicit maximum number of documents to return. Operations: `find` Requires version 4.26.0 or later. **Type**: `int` ### [](#operation)`operation` The mongodb operation to perform. Requires version 4.2.0 or later. **Type**: `string` **Default**: `find` **Options**: `find`, `aggregate` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#query)`query` Bloblang expression describing MongoDB query. **Type**: `string` ```yaml # Examples: query: |2 root.from = {"$lte": timestamp_unix()} root.to = {"$gte": timestamp_unix()} ``` ### [](#sort)`sort` An object specifying fields to sort by, and the respective sort order (`1` ascending, `-1` descending). Note: The driver currently appears to support only one sorting key. Operations: `find` Requires version 4.26.0 or later. **Type**: `object` ```yaml # Examples: sort: name: 1 # --- sort: age: -1 ``` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` --- # Page 63: mqtt **URL**: https://docs.redpanda.com/connect/components/inputs/mqtt.md --- # mqtt --- title: mqtt latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/mqtt page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mqtt.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mqtt.adoc description: Subscribe to topics on MQTT brokers. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Subscribe to topics on MQTT brokers. Introduced in version 4.37.0. #### Common ```yml input: label: "" mqtt: urls: [] # No default (required) client_id: "" connect_timeout: 30s topics: [] # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" mqtt: urls: [] # No default (required) client_id: "" dynamic_client_id_suffix: "" # No default (optional) connect_timeout: 30s will: enabled: false qos: 0 retained: false topic: "" payload: "" user: "" password: "" keepalive: 30 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topics: [] # No default (required) qos: 1 clean_session: true auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `mqtt_duplicate` - `mqtt_qos` - `mqtt_retained` - `mqtt_topic` - `mqtt_message_id` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#clean_session)`clean_session` Set whether the connection is non-persistent. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` The client identifier sent to the MQTT broker when connecting, which the broker uses to identify the client session. Use `dynamic_client_id_suffix` to give each run of the pipeline a unique identifier. **Type**: `string` **Default**: `""` ### [](#connect_timeout)`connect_timeout` The maximum amount of time to wait in order to establish a connection before the attempt is abandoned. Requires version 3.58.0 or later. **Type**: `string` **Default**: `30s` ```yaml # Examples: connect_timeout: 1s # --- connect_timeout: 500ms ``` ### [](#dynamic_client_id_suffix)`dynamic_client_id_suffix` Append a dynamically generated suffix to the specified `client_id` on each run of the pipeline. This can be useful when clustering Redpanda Connect producers. **Type**: `string` | Option | Summary | | --- | --- | | nanoid | append a nanoid of length 21 characters | ### [](#keepalive)`keepalive` Max seconds of inactivity before a keepalive message is sent. **Type**: `int` **Default**: `30` ### [](#password)`password` A password to connect with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#qos)`qos` The level of delivery guarantee to enforce. Has options 0, 1, 2. **Type**: `int` **Default**: `1` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topics)`topics[]` A list of topics to consume from. **Type**: `array` ### [](#urls)`urls[]` A list of URLs to connect to. Use the format `scheme://host:port`, where: - `scheme` is one of the following: `tcp`, `ssl`, `ws` - `host` is the IP address or hostname - `port` is the port on which the MQTT broker accepts connections If an item in the list contains commas, it is expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:1883" ``` ### [](#user)`user` A username to connect with. **Type**: `string` **Default**: `""` ### [](#will)`will` Set last will message in case of Redpanda Connect failure **Type**: `object` ### [](#will-enabled)`will.enabled` Whether to enable last will messages. **Type**: `bool` **Default**: `false` ### [](#will-payload)`will.payload` Set payload for last will message. **Type**: `string` **Default**: `""` ### [](#will-qos)`will.qos` Set QoS for last will message. Valid values are: 0, 1, 2. **Type**: `int` **Default**: `0` ### [](#will-retained)`will.retained` Set retained for last will message. **Type**: `bool` **Default**: `false` ### [](#will-topic)`will.topic` Set topic for last will message. **Type**: `string` **Default**: `""` --- # Page 64: mysql_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/mysql_cdc.md --- # mysql_cdc --- title: mysql_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/mysql_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mysql_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mysql_cdc.adoc description: Enables MySQL streaming for Redpanda Connect. page-git-created-date: "2025-02-20" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data changes from a MySQL database, using MySQL’s binary log to capture data updates. This input is built on the [`mysql-canal` library](https://github.com/go-mysql-org/go-mysql?tab=readme-ov-file#replication) but uses a custom approach for streaming historical data. Introduced in version 4.46.0. #### Common ```yml input: label: "" mysql_cdc: flavor: mysql dsn: "" # No default (required) tables: [] # No default (required) checkpoint_cache: "" # No default (required) checkpoint_key: mysql_binlog_position snapshot_max_batch_size: 1000 stream_snapshot: false # No default (required) max_parallel_snapshot_tables: 1 auto_replay_nacks: true checkpoint_limit: 1024 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml input: label: "" mysql_cdc: flavor: mysql dsn: "" # No default (required) tables: [] # No default (required) checkpoint_cache: "" # No default (required) checkpoint_key: mysql_binlog_position snapshot_max_batch_size: 1000 max_reconnect_attempts: 10 stream_snapshot: false # No default (required) max_parallel_snapshot_tables: 1 auto_replay_nacks: true checkpoint_limit: 1024 tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (required) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` The `mysql_cdc` input uses MySQL’s [binary log (`binlog`)](https://dev.mysql.com/doc/refman/8.0/en/binary-log.html) to capture changes made to a MySQL database in real time and streams them to Redpanda Connect. Redpanda Connect allows you to specify which [database tables](#tables) in your source database to receive changes from. There are also [two replication modes](#choose-a-replication-mode) to choose from. ## [](#prerequisites)Prerequisites - MySQL version 8 or later - A MySQL instance with binary logging enabled ### [](#configuration-resources)Configuration resources #### Cloud platforms - [Change data capture on Amazon RDS for MySQL](https://aws.amazon.com/blogs/database/enable-change-data-capture-on-amazon-rds-for-mysql-applications-that-are-using-xa-transactions/) - [Azure MySQL Database (CDC)](https://learn.microsoft.com/en-us/fabric/real-time-hub/add-source-mysql-database-cdc) - [Google Cloud SQL for MySQL](https://cloud.google.com/datastream/docs/configure-cloudsql-mysql) #### Self-hosted MySQL - [Binary Logging Options and Variables](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html) ## [](#choose-a-replication-mode)Choose a replication mode You can run the `mysql_cdc` input in one of two modes, depending on whether you need a snapshot of existing data. - Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected tables and streams the contents before processing changes from the last recorded binlog position. - Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest binlog position. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect connects to your MySQL database and does the following to capture a snapshot of all data in the selected tables: 1. Executes the `FLUSH TABLES WITH READ LOCK` query to write any outstanding table updates to disk, and locks the tables. 2. Runs the `START TRANSACTION WITH CONSISTENT SNAPSHOT` statement to create a new transaction with a consistent view of all data, capturing the state of the database at the moment the transaction started. 3. Reads the current binlog position. 4. Runs the `UNLOCK TABLES` statement to release the database. 5. Preserves the initial transaction for data integrity. > 📝 **NOTE** > > If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current binlog position in the [`checkpoint_cache`](#checkpoint_cache). After the snapshot is taken, the input executes SELECT statements to extract data from the selected tables in two stages: 1. The input finds the primary keys of a table. 2. It selects the data ordered by primary key. Finally, the input uses the stored binlog position to catch up with changes that occurred during snapshot processing. ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect connects to your MySQL database and starts processing data changes from the latest binlog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last binlog position written to the [`checkpoint_cache`](#checkpoint_cache). ## [](#binlog-rotation)Binlog rotation While the `mysql_cdc` input is streaming changes to Redpanda Connect, your MySQL server may rotate the binlog file. When this occurs, Redpanda Connect flushes the existing message batch and stores the new binlog position so that it can resume processing using the latest offset. ## [](#data-mappings)Data mappings The following table shows how selected MySQL data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values. | MySQL data type | Bloblang value | | --- | --- | | TEXT, VARCHAR | A string value, for example: "this data" | | BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB | An array of byte values, for example: [byte1,byte2,byte3] | | DECIMAL, NUMERIC, TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT, YEAR | A standard numeric type, for example: 123 | | FLOAT, DOUBLE | A 64-bit decimal (float64), for example: 123.1234 | | DATETIME, TIMESTAMP | A Bloblang timestamp, for example:1257894000000 2009-11-10 23:00:00 +0000 UTC | | SET | An array of strings, for example: ["apple", "banana", "orange"] | | JSON | A map object of the JSON, for example: {"red": 1, "blue": 2, "green": 3} | ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `operation`: The type of operation (insert, update, delete, or read for snapshot messages) - `table`: The name of the table - `binlog_position`: The binlog position (for CDC messages only, not set for snapshot messages) - `schema`: The table schema in benthos common schema format, compatible with processors like parquet\_encode ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication configuration for MySQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. Requires version 4.72.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication for MySQL. When enabled, an IAM authentication token is generated and used as the password. When using IAM authentication ensure `max_reconnect_attempts` is set to a low value to ensure it can refresh credentials. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` The MySQL endpoint hostname (for example, mydb.abc123.us-east-1.rds.amazonaws.com). **Type**: `string` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region where the MySQL instance is located. The region is used to sign the IAM authentication token and for STS calls when assuming roles. If no region is specified, the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. When `roles` is also set, this role is assumed first and the `roles` entries are assumed after it. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. When `role` is also set, it is assumed before the first entry. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) to store the binlog position of the most recent data update delivered by Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this last known position, avoiding the need to reprocess all table updates. **Type**: `string` ### [](#checkpoint_key)`checkpoint_key` The key identifier used to store the binlog position in `checkpoint_cache`. If you have multiple `mysql_cdc` inputs sharing the same cache, you can provide an alternative key. **Type**: `string` **Default**: `mysql_binlog_position` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given binlog position is not acknowledged until all messages up to that position are delivered. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` The data source name (DSN) of the MySQL database from which you want to stream updates. Use the format `user:password@tcp(localhost:3306)/database`. **Type**: `string` ```yaml # Examples: dsn: user:password@tcp(localhost:3306)/database ``` ### [](#flavor)`flavor` The type of MySQL database to connect to. Requires version 4.48.0 or later. **Type**: `string` **Default**: `mysql` | Option | Summary | | --- | --- | | mariadb | MariaDB flavored databases. | | mysql | MySQL flavored databases. | ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specifies the number of tables that will be snapshotted in parallel. Requires version 4.90.0 or later. **Type**: `int` **Default**: `1` ### [](#max_reconnect_attempts)`max_reconnect_attempts` The maximum number of attempts the MySQL driver will try to re-establish a broken connection before Connect attempts reconnection. A zero or negative number means infinite retry attempts. Requires version 4.72.0 or later. **Type**: `int` **Default**: `10` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of table rows to fetch in each batch when taking a snapshot. This option is only available when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1000` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source database before streaming data changes. To use this setting, all database tables that you want to replicate _must_ have a primary key. When set to `false`, the input starts streaming from the current binlog position. **Type**: `bool` ### [](#tables)`tables[]` A list of the database table names to stream changes from. Specify each table name as a separate item. **Type**: `array` ```yaml # Examples: tables: - table1 - table2 ``` ### [](#tls)`tls` Custom TLS settings for the MySQL connection. When `enabled` is `true`, these settings replace any `tls` parameter in the `dsn`, and the server name is set to the host from the DSN. Requires version 4.72.0 or later. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 65: nanomsg **URL**: https://docs.redpanda.com/connect/components/inputs/nanomsg.md --- # nanomsg --- title: nanomsg latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/nanomsg page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nanomsg.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nanomsg.adoc description: Consumes messages via Nanomsg sockets (scalability protocols). page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes messages via Nanomsg sockets (scalability protocols). Currently only PULL and SUB sockets are supported. #### Common ```yml input: label: "" nanomsg: urls: [] # No default (required) bind: true socket_type: PULL auto_replay_nacks: true sub_filters: [] ``` #### Advanced ```yml input: label: "" nanomsg: urls: [] # No default (required) bind: true socket_type: PULL auto_replay_nacks: true sub_filters: [] poll_timeout: 5s ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#bind)`bind` Whether the URLs provided should be connected to, or bound as. **Type**: `bool` **Default**: `true` ### [](#poll_timeout)`poll_timeout` The period to wait until a poll is abandoned and reattempted. **Type**: `string` **Default**: `5s` ### [](#socket_type)`socket_type` The socket type to use. **Type**: `string` **Default**: `PULL` **Options**: `PULL`, `SUB` ### [](#sub_filters)`sub_filters[]` A list of topic prefixes to subscribe to when `socket_type` is `SUB`. A message is received only when its body starts with one of the prefixes, so a single empty filter (`''`) subscribes to everything. A SUB socket requires at least one filter. **Type**: `array` **Default**: `[]` ### [](#urls)`urls[]` A list of URLs to connect to, or to listen on when `bind` is `true`. If an item in the list contains commas, it is split into multiple URLs. A **host, such as `tcp://`**`:5555`, is replaced with `0.0.0.0`. **Type**: `array` --- # Page 66: nats_jetstream **URL**: https://docs.redpanda.com/connect/components/inputs/nats_jetstream.md --- # nats_jetstream --- title: nats_jetstream latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/nats_jetstream page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats_jetstream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats_jetstream.adoc description: Reads messages from NATS JetStream subjects. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads messages from NATS JetStream subjects. Introduced in version 3.46.0. #### Common ```yml input: label: "" nats_jetstream: urls: [] # No default (required) queue: "" # No default (optional) subject: "" # No default (optional) durable: "" # No default (optional) stream: "" # No default (optional) bind: false # No default (optional) deliver: all ``` #### Advanced ```yml input: label: "" nats_jetstream: urls: [] # No default (required) max_reconnects: 0 # No default (optional) queue: "" # No default (optional) subject: "" # No default (optional) durable: "" # No default (optional) stream: "" # No default (optional) bind: false # No default (optional) create_stream: false deliver: all ack_wait: 30s max_ack_pending: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#consume-mirrored-streams)Consume mirrored streams When a stream being consumed is mirrored in a different JetStream domain, the stream cannot be resolved from the subject name alone. You must specify the stream name as well as the subject (if applicable). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_subject` - `nats_sequence_stream` - `nats_sequence_consumer` - `nats_num_delivered` - `nats_num_pending` - `nats_domain` - `nats_timestamp_unix_nano` - `nats_consumer` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#ack_wait)`ack_wait` The maximum amount of time NATS server should wait for an ack from consumer. **Type**: `string` **Default**: `30s` ```yaml # Examples: ack_wait: 100ms # --- ack_wait: 5m ``` ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#bind)`bind` Indicates that the subscription should use an existing consumer. **Type**: `bool` ### [](#create_stream)`create_stream` Whether to automatically create the stream if it doesn’t exist (requires the stream field to be set). Requires version 4.84.0 or later. **Type**: `bool` **Default**: `false` ### [](#deliver)`deliver` Determines which messages to deliver when consuming without a durable subscriber. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Deliver all available messages. | | last | Deliver starting with the last published messages. | | last_per_subject | Deliver starting with the last published message per subject. | | new | Deliver starting from now, not taking into account any previous messages. | ### [](#durable)`durable` Preserve the state of your consumer under a durable name. **Type**: `string` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_ack_pending)`max_ack_pending` The maximum number of outstanding acks to be allowed before consuming is halted. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#queue)`queue` An optional queue group to consume as. Used to configure a push consumer. **Type**: `string` ### [](#stream)`stream` A stream to consume from. Either a subject or stream must be specified. **Type**: `string` ### [](#subject)`subject` A subject to consume from. Supports wildcards for consuming multiple subjects. Either a subject or stream must be specified. **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: foo.*.baz # --- subject: foo.bar.* # --- subject: foo.> ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 67: nats_kv **URL**: https://docs.redpanda.com/connect/components/inputs/nats_kv.md --- # nats_kv --- title: nats_kv latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/nats_kv page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats_kv.adoc description: Watches for updates in a NATS key-value bucket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Watches for updates in a NATS key-value bucket. Introduced in version 4.12.0. #### Common ```yml input: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) key: ">" auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" nats_kv: urls: [] # No default (required) max_reconnects: 0 # No default (optional) bucket: "" # No default (required) key: ">" auto_replay_nacks: true ignore_deletes: false include_history: false meta_only: false tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_kv_key` - `nats_kv_bucket` - `nats_kv_revision` - `nats_kv_delta` - `nats_kv_operation` - `nats_kv_created` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#ignore_deletes)`ignore_deletes` Do not send delete markers as messages. **Type**: `bool` **Default**: `false` ### [](#include_history)`include_history` Include all the history per key, not just the last one. **Type**: `bool` **Default**: `false` ### [](#key)`key` Key to watch for updates, can include wildcards. **Type**: `string` **Default**: `>` ```yaml # Examples: key: foo.bar.baz # --- key: foo.*.baz # --- key: foo.bar.* # --- key: foo.> ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#meta_only)`meta_only` Retrieve only the metadata of the entry **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 68: nats_stream **URL**: https://docs.redpanda.com/connect/components/inputs/nats_stream.md --- # nats_stream --- title: nats_stream latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/nats_stream page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats_stream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats_stream.adoc description: Subscribe to a NATS Stream subject. Joining a queue is optional and allows multiple clients of a subject to consume using queue semantics. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Subscribe to a NATS Stream subject. Joining a queue is optional and allows multiple clients of a subject to consume using queue semantics. #### Common ```yml input: label: "" nats_stream: urls: [] # No default (required) cluster_id: "" # No default (required) client_id: "" queue: "" subject: "" durable_name: "" unsubscribe_on_close: false ``` #### Advanced ```yml input: label: "" nats_stream: urls: [] # No default (required) max_reconnects: 0 # No default (optional) cluster_id: "" # No default (required) client_id: "" queue: "" subject: "" durable_name: "" unsubscribe_on_close: false start_from_oldest: true max_inflight: 1024 ack_wait: 30s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` > ⚠️ **CAUTION: Deprecation notice** > > Deprecation notice > > The NATS Streaming Server is being deprecated. Critical bug fixes and security fixes will be applied until June of 2023. NATS-enabled applications requiring persistence should use [JetStream](https://docs.nats.io/nats-concepts/jetstream). Tracking and persisting offsets through a durable name is also optional and works with or without a queue. If a durable name is not provided then subjects are consumed from the most recently published message. When a consumer closes its connection, it unsubscribes from the subject. When all consumers of a durable queue do this, the offsets are deleted from the NATS server. To avoid this, set the `unsubscribe_on_close` field to `false`, to prevent consumers from unsubscribing. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_stream_subject` - `nats_stream_sequence` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication There are a number of Redpanda Connect components that use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#ack_wait)`ack_wait` An optional duration to specify at which a message that is yet to be acked will be automatically retried. **Type**: `string` **Default**: `30s` ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#client_id)`client_id` A client ID to connect as. **Type**: `string` **Default**: `""` ### [](#cluster_id)`cluster_id` The ID of the cluster to consume from. **Type**: `string` ### [](#durable_name)`durable_name` Preserve the state of your consumer under a durable name. **Type**: `string` **Default**: `""` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_inflight)`max_inflight` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#queue)`queue` The queue to consume from. **Type**: `string` **Default**: `""` ### [](#start_from_oldest)`start_from_oldest` If a position is not found for a queue, determines whether to consume from the oldest available message, otherwise messages are consumed from the latest. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` A subject to consume from. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#unsubscribe_on_close)`unsubscribe_on_close` Whether the subscription should be destroyed when this client disconnects. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 69: nats **URL**: https://docs.redpanda.com/connect/components/inputs/nats.md --- # nats --- title: nats latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/nats page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats.adoc description: Subscribe to a NATS subject. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Subscribe to a NATS subject. #### Common ```yml input: label: "" nats: urls: [] # No default (required) subject: "" # No default (required) queue: "" # No default (optional) auto_replay_nacks: true send_ack: true ``` #### Advanced ```yml input: label: "" nats: urls: [] # No default (required) max_reconnects: 0 # No default (optional) subject: "" # No default (required) queue: "" # No default (optional) auto_replay_nacks: true send_ack: true nak_delay: "" # No default (optional) prefetch_count: 500000 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_subject` - `nats_reply_subject` - All message headers (when supported by the connection) You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#nak_delay)`nak_delay` An optional delay duration on redelivering a message when negatively acknowledged. Requires version 4.16.0 or later. **Type**: `string` ```yaml # Examples: nak_delay: 1m ``` ### [](#prefetch_count)`prefetch_count` The maximum number of messages to pull at a time. **Type**: `int` **Default**: `500000` ### [](#queue)`queue` An optional queue group to consume as. **Type**: `string` ### [](#send_ack)`send_ack` Whether an automatic acknowledgment is sent as a reply to each message. When enabled, these replies are sent only when data has been delivered to all outputs. Requires version 4.33.0 or later. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` A subject to consume from. Supports wildcards for consuming multiple subjects. **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: foo.*.baz # --- subject: foo.bar.* # --- subject: foo.> ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 70: nsq **URL**: https://docs.redpanda.com/connect/components/inputs/nsq.md --- # nsq --- title: nsq latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/nsq page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nsq.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nsq.adoc description: Subscribe to an NSQ instance topic and channel. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Subscribe to an NSQ instance topic and channel. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nsq_attempts` - `nsq_id` - `nsq_nsqd_address` - `nsq_timestamp` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" nsq: nsqd_tcp_addresses: [] # No default (required) lookupd_http_addresses: [] # No default (required) topic: "" # No default (required) channel: "" # No default (required) user_agent: "" # No default (optional) max_in_flight: 100 max_attempts: 5 ``` ### Advanced ```yml input: label: "" nsq: nsqd_tcp_addresses: [] # No default (required) lookupd_http_addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topic: "" # No default (required) channel: "" # No default (required) user_agent: "" # No default (optional) max_in_flight: 100 max_attempts: 5 ``` ## [](#fields)Fields ### [](#channel)`channel` The channel to consume from. **Type**: `string` ### [](#lookupd_http_addresses)`lookupd_http_addresses[]` A list of nsqlookupd addresses to connect to. **Type**: `array` ### [](#max_attempts)`max_attempts` The maximum number of attempts to successfully consume a messages. Requires version 4.11.0 or later. **Type**: `int` **Default**: `5` ### [](#max_in_flight)`max_in_flight` The maximum number of pending messages to consume at any given time. **Type**: `int` **Default**: `100` ### [](#nsqd_tcp_addresses)`nsqd_tcp_addresses[]` A list of nsqd addresses to connect to. **Type**: `array` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to consume from. **Type**: `string` ### [](#user_agent)`user_agent` A user agent to assume when connecting. **Type**: `string` --- # Page 71: ockam_kafka **URL**: https://docs.redpanda.com/connect/components/inputs/ockam_kafka.md --- # ockam_kafka --- title: ockam_kafka latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/ockam_kafka page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/ockam_kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/ockam_kafka.adoc description: Uses Ockam to decrypt and read end-to-end encrypted messages from Kafka topics. page-git-created-date: "2024-11-25" page-git-modified-date: "2026-10-10" --- Uses [Ockam](https://docs.ockam.io/) to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the `ockam_kafka` output or by [creating a Kafka Portal Inlet](https://command.ockam.io/manual/ockam-kafka-inlet-create.html) using Ockam Command. [Ockam Secure Channels](https://docs.ockam.io/reference/command/secure-channels) guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages. > 📝 **NOTE** > > You can use Ockam to encrypt whole messages, or specific fields in a message. Introduced in version 4.33.0. #### Common ```yml input: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_producer: self relay: "" # No default (optional) node_address: 127.0.0.1:6262 encrypted_fields: [] ``` #### Advanced ```yml input: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) checkpoint_limit: 1024 commit_period: 5s multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) topic_lag_refresh_period: 5s disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_producer: self relay: "" # No default (optional) node_address: 127.0.0.1:6262 encrypted_fields: [] ``` ## [](#fields)Fields ### [](#allow)`allow` Use in conjunction with the `route_to_kafka_outlet` field to specify an access control policy for the Kafka Portal Outlet. For example, setting this value to `kafka_us_east` forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute `kafka_us_east=true`. **Type**: `string` **Default**: `self` ### [](#allow_producer)`allow_producer` Specify an access control policy for producers. For example, setting this value to `orders_producer` forces the producer to present an Ockam credential, which confirms that the producer has the attribute `orders_producer=true`. **Type**: `string` **Default**: `self` ### [](#disable_content_encryption)`disable_content_encryption` Disables encryption of Kafka message payloads. If this value is set to `true`: - Only message payloads are unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled. - All other `ockam_kafka` inputs and outputs that use the same topic must also set this field to `true`. **Type**: `bool` **Default**: `false` ### [](#encrypted_fields)`encrypted_fields[]` The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted. Requires version 4.37.0 or later. **Type**: `array` **Default**: `[]` ### [](#enrollment_ticket)`enrollment_ticket` The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional). You can generate a new ticket using the [`ockam project ticket` command](https://command.ockam.io/manual/ockam-project-ticket.html). **Type**: `string` ### [](#identity_name)`identity_name` The name of the [Ockam identity](https://command.ockam.io/manual/ockam-identity.html) to use (optional). If an identity with this name does not exist, it is created. If this value is not provided, the default Ockam identity is used, and it is created if it does not exist. **Type**: `string` ### [](#kafka)`kafka` **Type**: `object` ### [](#kafka-batching)`kafka.batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/) that applies to individual topic partitions, so that the input batches messages together before flushing them for processing. Batching can improve performance and is useful for windowed processing. Batching this way preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#kafka-batching-byte_size)`kafka.batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#kafka-batching-check)`kafka.batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#kafka-batching-count)`kafka.batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#kafka-batching-period)`kafka.batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#kafka-batching-processors)`kafka.batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#kafka-checkpoint_limit)`kafka.checkpoint_limit` The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value. **Type**: `int` **Default**: `1024` ### [](#kafka-commit_period)`kafka.commit_period` The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#kafka-consumer_group)`kafka.consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics specified in the `topics` field are automatically distributed across consumers that share the consumer group. - Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#kafka-fetch_max_bytes)`kafka.fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. This field is equivalent to the Java setting `fetch.max.bytes`. Requires version 4.39.0 or later. **Type**: `string` **Default**: `50MiB` ### [](#kafka-fetch_max_partition_bytes)`kafka.fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. Requires version 4.39.0 or later. **Type**: `string` **Default**: `1MiB` ### [](#kafka-fetch_max_wait)`kafka.fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). This field is equivalent to the Java setting `fetch.max.wait.ms`. Requires version 4.45.0 or later. **Type**: `string` **Default**: `5s` ### [](#kafka-fetch_min_bytes)`kafka.fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. Requires version 4.39.0 or later. **Type**: `string` **Default**: `1B` ### [](#kafka-heartbeat_interval)`kafka.heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. This value must be lower than `session_timeout`, and should be no higher than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `3s` ### [](#kafka-instance_id)`kafka.instance_id` When you specify a `consumer_group`, set `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. The value must be unique per consumer within the group. When you assign an instance ID, the client does not leave the consumer group when it closes. To remove the client from the group, you must use an external admin command on behalf of the instance ID. Requires version 4.46.0 or later. **Type**: `string` **Default**: `""` ### [](#kafka-multi_header)`kafka.multi_header` Decode headers into lists to allow the handling of multiple values with the same key. **Type**: `bool` **Default**: `false` ### [](#kafka-rack_id)`kafka.rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#kafka-rebalance_timeout)`kafka.rebalance_timeout` When you specify a `consumer_group`, `rebalance_timeout` sets how long consumer group members can take to complete their work and commit offsets after a rebalance has begun. The time that a member takes to detect the rebalance (from a heartbeat) counts against this timeout. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `45s` ### [](#kafka-regexp_topics_exclude)`kafka.regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. Requires version 4.69.0 or later. **Type**: `array` ### [](#kafka-regexp_topics_include)`kafka.regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. Requires version 4.69.0 or later. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#kafka-seed_brokers)`kafka.seed_brokers[]` The address of the Kafka broker that the Ockam node’s Kafka Outlet connects to. This field is only used, and is then required, when `route_to_kafka_outlet` is set to `self`. Only one address is used: the first list item, up to its first comma. If the list contains more than one item, a warning is logged. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" ``` ### [](#kafka-session_timeout)`kafka.session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `1m` ### [](#kafka-start_offset)`kafka.start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. Requires version 4.53.0 or later. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#kafka-tls)`kafka.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#kafka-tls-client_certs)`kafka.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#kafka-tls-client_certs-cert)`kafka.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-cert_file)`kafka.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key)`kafka.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key_file)`kafka.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-password)`kafka.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#kafka-tls-enable_renegotiation)`kafka.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-enabled)`kafka.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-root_cas)`kafka.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#kafka-tls-root_cas_file)`kafka.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#kafka-tls-skip_cert_verify)`kafka.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#kafka-topic_lag_refresh_period)`kafka.topic_lag_refresh_period` The interval between consumer lag refreshes. During each cycle, this input asks the brokers for the consumer group’s committed offsets and the partition end offsets, and records the difference (the number of unread messages) for each topic partition in the consumer lag metric and the `kafka_lag` metadata field. Lag is only refreshed when `consumer_group` is set. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.54.1 or later. **Type**: `string` **Default**: `5s` ### [](#kafka-topics)`kafka.topics[]` A list of topics to consume from. You can list multiple comma-separated topics in a single element. If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusive. Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#kafka-transaction_isolation_level)`kafka.transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. Requires version 4.53.0 or later. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ### [](#node_address)`node_address` The TCP listening address of the Ockam node. **Type**: `string` **Default**: `127.0.0.1:6262` ### [](#relay)`relay` Make the Ockam node accessible through a relay with the supplied name (optional). This field only takes effect when `enrollment_ticket` is set. For example, setting this value to `orders_consumer` requires you to set `route_to_consumer` on any producer to `/project/default/service/forward_to_orders_consumer/secure/api`. **Type**: `string` ### [](#route_to_kafka_outlet)`route_to_kafka_outlet` The route to reach the Kafka Portal Outlet of your Ockam portal. For example, `/project/default`. **Type**: `string` **Default**: `self` --- # Page 72: oracledb_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/oracledb_cdc.md --- # oracledb_cdc --- title: oracledb_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/oracledb_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/oracledb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/oracledb_cdc.adoc description: Enables Change Data Capture by consuming from OracleDB. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Enables Change Data Capture by consuming from OracleDB. Introduced in version 4.83.0. Streams changes from an Oracle database for Change Data Capture (CDC). Use the `snapshot_mode` field to control whether existing data is captured in an initial snapshot before streaming changes. #### Common ```yml input: label: "" oracledb_cdc: connection_string: "" # No default (required) wallet_path: "" # No default (optional) wallet_password: "" # No default (optional) prefetch_rows: 500 snapshot_mode: "" # No default (optional) max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 logminer: scn_window_size: 20000 min_scn_window_size: 1000 max_scn_window_size: 100000 backoff_interval: 5s mining_interval: 300ms strategy: online_catalog max_transaction_events: 0 lob_enabled: true transaction_cache: "" # No default (optional) transaction_cache_key: oracledb_cdc max_session_age: 0s snapshot_filters: {} # No default (optional) include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE checkpoint_cache_key: oracledb_cdc checkpoint_limit: 1024 pdb_name: "" # No default (optional) auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml input: label: "" oracledb_cdc: connection_string: "" # No default (required) wallet_path: "" # No default (optional) wallet_password: "" # No default (optional) prefetch_rows: 500 snapshot_mode: "" # No default (optional) max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 logminer: scn_window_size: 20000 min_scn_window_size: 1000 max_scn_window_size: 100000 window_strategy: scn_window redo_volume_min: 2 redo_volume_growth_max: 4 backoff_interval: 5s mining_interval: 300ms strategy: online_catalog max_transaction_events: 0 lob_enabled: true transaction_cache: "" # No default (optional) transaction_cache_key: oracledb_cdc max_session_age: 0s snapshot_filters: {} # No default (optional) include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE checkpoint_cache_key: oracledb_cdc checkpoint_limit: 1024 pdb_name: "" # No default (optional) auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `database_schema`: The database schema for the table where the message originates from. - `table_name`: Name of the table that the message originated from. - `operation`: Type of operation that generated the message: "read", "delete", "insert", or "update". "read" is from messages that are read in the initial snapshot phase. - `scn`: The System Change Number in Oracle. Messages published as part of a snapshot will contain Oracle’s current SCN captured at time of snapshot. - `transaction_id`: The Oracle transaction ID in `USN.SLOT.SEQ` format, identifying the transaction that produced the change. Not present on snapshot (`read`) messages. - `source_ts_ms`: The timestamp of when Oracle wrote the change record into the redo log, expressed as milliseconds since the Unix epoch. This reflects the database server’s wall-clock time at the moment the DML executed, not the transaction commit time. Not present on snapshot (`read`) messages. - `commit_ts_ms`: The timestamp of the transaction commit, expressed as milliseconds since the Unix epoch. Sourced from `V$LOGMNR_CONTENTS.TIMESTAMP` on the COMMIT redo record: this is Oracle’s wall-clock time when the commit was written to the redo log, not a dedicated commit-timestamp column. For snapshot (`read`) messages, this reflects Oracle’s `SYSTIMESTAMP` at the moment the snapshot SCN was captured, so all snapshot messages share the same value. - `username`: The Oracle database username of the session that performed the DML, sourced from `V$LOGMNR_CONTENTS.USERNAME`. Not present on snapshot (`read`) messages, nor on change messages where Oracle reports a NULL or empty username. - `rs_id`: The LogMiner record set ID from `V$LOGMNR_CONTENTS.RS_ID`, identifying the redo record that produced the change. Use `rs_id`, `ssn` and `row_seq` together to identify one row change and to order the changes of a transaction. Not present on snapshot (`read`) messages, nor on LOB-only synthetic updates. - `ssn`: The SQL sequence number from `V$LOGMNR_CONTENTS.SSN`. Compare it numerically. Not present whenever `rs_id` is not present. - `row_seq`: The position of the row change among the consecutive row changes that report the same `rs_id` and `ssn`, starting at 0 in redo order. Oracle reports the same `rs_id` and `ssn` for every row of an array DML redo record, such as a bulk `DELETE` of up to 255 rows, so this field tells those rows apart. It is 0 for every other change. LogMiner returns the rows of one redo record one after the other, and the connector counts them in that order. Compare it numerically. Not present whenever `rs_id` is not present. - `schema`: The table schema, for use with schema-aware downstream processors such as `schema_registry_encode`. When new columns are detected in CDC events, the schema is automatically refreshed from the Oracle catalog. Dropped columns are reflected after a connector restart. > 📝 **NOTE** > > `source_ts_ms` is not present on snapshot (`read`) messages. ## [](#permissions)Permissions When using the default Oracle-based cache, the Connect user requires permission to create tables and stored procedures, and the rpcn schema must already exist. See `checkpoint_cache_table_name` for more information. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) to use for storing the current System Change Number (SCN) that has been successfully delivered. This allows Redpanda Connect to continue from that SCN upon restart, rather than consume the entire state of OracleDB redo logs. If not set, the default Oracle-based cache is used. See `checkpoint_cache_table_name` for more information. **Type**: `string` ### [](#checkpoint_cache_key)`checkpoint_cache_key` The key to use to store the snapshot position in `checkpoint_cache`. An alternative key can be provided if multiple CDC inputs share the same cache. **Type**: `string` **Default**: `oracledb_cdc` ### [](#checkpoint_cache_table_name)`checkpoint_cache_table_name` The identifier for the checkpoint cache table name. If no `checkpoint_cache` field is specified, this input will automatically create a table and stored procedure under the `rpcn` schema to act as a checkpoint cache. This table stores the latest processed System Change Number (SCN) that has been successfully delivered, allowing Redpanda Connect to resume from that point upon restart rather than reconsume the entire redo log. When `pdb_name` is set and this field is left at its default value, the table name is automatically derived per PDB (for example, `RPCN.CDC_CHECKPOINT_MYPDB`) to avoid SCN collisions between pipelines monitoring different PDBs. Set this field explicitly to opt out of that auto-derivation. **Type**: `string` **Default**: `RPCN.CDC_CHECKPOINT_CACHE` ```yaml # Examples: checkpoint_cache_table_name: RPCN.CHECKPOINT_CACHE ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level. Any given System Change Number (SCN) will not be acknowledged unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#connection_string)`connection_string` The connection string of the Oracle database to connect to. You can supply additional connection options as URL query parameters, for example: `oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true`. **Type**: `string` ```yaml # Examples: connection_string: oracle://username:password@host:port/service_name # --- connection_string: oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true ``` ### [](#exclude)`exclude[]` Regular expressions for tables to exclude. **Type**: `array` ```yaml # Examples: exclude: SCHEMA.PRIVATETABLE ``` ### [](#include)`include[]` Regular expressions for tables to include. **Type**: `array` ```yaml # Examples: include: SCHEMA.PRODUCTS ``` ### [](#logminer)`logminer` LogMiner configuration settings. **Type**: `object` ### [](#logminer-backoff_interval)`logminer.backoff_interval` The interval between attempts to check for new changes once all data is processed. For low traffic tables increasing this value can reduce network traffic to the server. **Type**: `string` **Default**: `5s` ```yaml # Examples: backoff_interval: 5s # --- backoff_interval: 1m ``` ### [](#logminer-lob_enabled)`logminer.lob_enabled` When enabled, large object (CLOB, BLOB) columns are included in both snapshot and streaming change events. When disabled, these columns are still present but contain no values. Enabling this option introduces additional performance overhead and increases memory requirements. Requires version 4.84.1 or later. **Type**: `bool` **Default**: `true` ### [](#logminer-max_scn_window_size)`logminer.max_scn_window_size` The maximum SCN range that can be mined in a single cycle. The window starts at scn\_window\_size and grows by scn\_window\_size each cycle that ends at the cap (backlog present), up to this limit. It shrinks by the same step each cycle that catches up to the database. This allows the connector to automatically mine larger windows during heavy backlog and smaller windows during steady state. Requires version 4.97.0 or later. **Type**: `int` **Default**: `100000` ### [](#logminer-max_session_age)`logminer.max_session_age` The maximum duration a single LogMiner session may stay open before being forcibly ended and restarted, even if the underlying redo log files haven’t changed. By default, a LogMiner session is only restarted when a redo log switch is detected. On databases where switches are infrequent, a session can stay open for a long time, and LogMiner has been observed to accumulate server-side PGA memory (particularly around online catalog dictionary lookups) until Oracle terminates the session with ORA-04036. Setting this forces a periodic restart independent of log switches. Set to 0 (default) to disable and restart only on log switches. Requires version 4.103.2 or later. **Type**: `string` **Default**: `0s` ```yaml # Examples: max_session_age: 20m ``` ### [](#logminer-max_transaction_events)`logminer.max_transaction_events` The maximum number of events that can be buffered for a single transaction. If a transaction exceeds this limit it is discarded and its events will not be emitted. Set to 0 to disable the limit. **Type**: `int` **Default**: `0` ### [](#logminer-min_scn_window_size)`logminer.min_scn_window_size` The minimum SCN gap required before starting a new LogMiner session. When the gap between the connector’s current position and the database’s current SCN is smaller than this value, the mining cycle is skipped and the connector backs off instead. This prevents excessive LogMiner start/stop cycles on low-traffic databases where Oracle background activity advances the SCN without producing relevant events. This gate applies regardless of `window_strategy`. Set to 0 to disable. Requires version 4.97.0 or later. **Type**: `int` **Default**: `1000` ### [](#logminer-mining_interval)`logminer.mining_interval` The interval between mining cycles during normal operation. Controls how frequently LogMiner polls for new changes when not caught up. **Type**: `string` **Default**: `300ms` ```yaml # Examples: mining_interval: 100ms # --- mining_interval: 1s ``` ### [](#logminer-redo_volume_growth_max)`logminer.redo_volume_growth_max` The ceiling the per-thread redo-volume budget can grow to. The ceiling applies independently to each open redo thread, so on Real Application Clusters (RAC) configurations the total volume mined per cycle scales with the number of threads. The budget starts at `redo_volume_min` and grows automatically whenever something prevents the mining window from advancing, such as a long-running transaction holding it in place or a redo log being recycled mid-query, up to this limit. Only applies when `window_strategy` is `redo_volume`. **Type**: `int` **Default**: `4` ### [](#logminer-redo_volume_min)`logminer.redo_volume_min` The approximate minimum number of redo logs to read per redo thread in each mining cycle. Consider increasing this value if redo logs are small and rotate frequently, decreasing if redo logs are very large. Only applies when `window_strategy` is `redo_volume`. **Type**: `int` **Default**: `2` ### [](#logminer-scn_window_size)`logminer.scn_window_size` The SCN range to mine per cycle. Each cycle reads changes between the current SCN and current SCN + scn\_window\_size. Smaller values mean more frequent queries with lower memory usage but higher overhead; larger values reduce query frequency and improve throughput at the cost of higher memory usage per cycle. **Type**: `int` **Default**: `20000` ### [](#logminer-strategy)`logminer.strategy` Controls how LogMiner retrieves data dictionary information. `online_catalog` uses the current data dictionary for best performance but cannot capture DDL changes. Currently, only `online_catalog` is supported. **Type**: `string` **Default**: `online_catalog` ### [](#logminer-transaction_cache)`logminer.transaction_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) to use for buffering in-flight transactions. When set, DML events are serialized and stored in the named cache rather than held in memory, reducing connector memory usage for workloads with large or long-running transactions. If not set, an in-memory buffer is used. Each in-flight transaction is stored as N+1 cache entries: one metadata key holding the transaction ID, start SCN, and event count; and one event key per DML event. A transaction with 1000 events occupies 1001 cache entries. Each AddEvent call writes exactly two keys regardless of how many events the transaction has already accumulated. This cache is designed for low-latency stores with cheap per-operation cost. Redis and Memcached are the recommended backends. The built-in `memory:{}` cache works but provides no durability across restarts. High-latency or per-request-cost stores such as S3 or DynamoDB are not recommended. A transaction with 1000 events generates approximately 3000 cache operations across its lifetime, and because LogMiner processes events on a single goroutine, per-call latency directly reduces throughput. A backend that causes timeouts or errors will also cause the mining cycle to restart from an earlier checkpoint SCN, which can result in duplicate event delivery. Requires version 4.94.0 or later. **Type**: `string` ### [](#logminer-transaction_cache_key)`logminer.transaction_cache_key` The key prefix used when storing transactions in `transaction_cache`. An alternative prefix must be set if multiple `oracledb_cdc` inputs share the same cache resource, since Oracle transaction IDs (USN.SLOT.SEQ) are only unique within a single Oracle instance and would otherwise collide. Requires version 4.94.0 or later. **Type**: `string` **Default**: `oracledb_cdc` ### [](#logminer-window_strategy)`logminer.window_strategy` Controls how the SCN range mined per cycle is sized: - `scn_window` (default): Grows and shrinks a fixed SCN increment (`scn_window_size`) based on its backlog, bounded by `min_scn_window_size` and `max_scn_window_size`. - `redo_volume`: Sizes the range by a fixed redo-volume budget calculated by log size (configured by `redo_volume_min` and `redo_volume_growth_max`), independent of raw SCN movement. This can be helpful when a database’s SCN can advance without matching real transaction volume (for example, a Multitenant Container Database (CDB) shared SCN bumped by another Pluggable Database (PDB)), since `scn_window` would otherwise burn cycles growing its window over mostly-empty ranges, while `redo_volume` sizes each cycle by the redo it actually reads. On Real Application Clusters (RAC), the `redo_volume` budget applies per open thread, so volume mined per cycle scales with the number of open threads. **Type**: `string` **Default**: `scn_window` **Options**: `scn_window`, `redo_volume` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` The maximum number of tables to read in parallel during the initial snapshot. Each table is read by its own reader. **Type**: `int` **Default**: `1` ### [](#pdb_name)`pdb_name` The name of the pluggable database (PDB) to monitor. When connecting to a CDB root, LogMiner output is scoped to this PDB via SRC\_CON\_NAME filtering and catalog queries use ALTER SESSION SET CONTAINER to switch context. Requires GRANT SET CONTAINER TO CONTAINER=ALL. Requires version 4.88.0 or later. **Type**: `string` ### [](#prefetch_rows)`prefetch_rows` The number of rows fetched per network round-trip, for both snapshot and streaming reads. Higher values mean fewer round-trips but more memory per fetch, for each table snapshotted in parallel. A `PREFETCH_ROWS` query parameter in `connection_string` takes precedence. Requires version 4.111.0 or later. **Type**: `int` **Default**: `500` ### [](#snapshot_filters)`snapshot_filters` A map of fully-qualified table names (for example `SCHEMA.TABLE`) to SQL `SELECT` queries that override the default snapshot query for each table. Use this to filter or shape the rows captured during the initial snapshot. Each query must project every column of the table’s primary key (all columns of a composite key), even if it otherwise selects only a subset of columns. During a snapshot, Redpanda Connect pages through a table’s rows by filtering and sorting on the full primary key against the query’s own result set. If a primary key column isn’t projected, the snapshot fails part-way through, after the first batch of rows is read. Requires version 4.101.0 or later. **Type**: `object` ```yaml # Examples: snapshot_filters: TESTDB.PRODUCTS: SELECT * FROM TESTDB.PRODUCTS WHERE ID > 1000 TESTDB.USERS: SELECT * FROM TESTDB.USERS ``` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of rows fetched per query when taking a snapshot of a table with a `snapshot_filters` entry configured. Tables without one are streamed through a single unordered cursor, where this value only paces how often a cancellation is checked. **Type**: `int` **Default**: `1000` ### [](#snapshot_mode)`snapshot_mode` Controls snapshot behavior. `none` (default) skips snapshotting and starts streaming from the current SCN. `snapshot_only` performs a full snapshot, persists the SCN checkpoint, then stops without streaming. `snapshot_and_stream` performs a full snapshot then transitions to streaming. Requires version 4.99.0 or later. **Type**: `string` **Options**: `none`, `snapshot_only`, `snapshot_and_stream` ### [](#wallet_password)`wallet_password` Password for the `ewallet.p12` PKCS#12 wallet file. Only use this when the wallet directory contains `ewallet.p12` rather than `cwallet.sso`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.87.0 or later. **Type**: `string` ### [](#wallet_path)`wallet_path` Path to the Oracle Wallet directory. When set, this automatically enables SSL. The directory must contain either `cwallet.sso` (auto-login, does not require a password) or `ewallet.p12` (requires `wallet_password`). Requires version 4.87.0 or later. **Type**: `string` ```yaml # Examples: wallet_path: /opt/oracle/wallet ``` --- # Page 73: otlp_grpc **URL**: https://docs.redpanda.com/connect/components/inputs/otlp_grpc.md --- # otlp_grpc --- title: otlp_grpc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/otlp_grpc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/otlp_grpc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/otlp_grpc.adoc description: Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Introduced in version 4.78.0. Exposes an OpenTelemetry Collector gRPC receiver that accepts traces, logs, and metrics via gRPC. Telemetry data is received in OTLP protobuf format and converted to individual Redpanda OTEL v1 protobuf messages. Each signal (span, log record, or metric) becomes a separate message with embedded Resource and Scope metadata, optimized for Kafka partitioning. #### Common ```yml input: label: "" otlp_grpc: encoding: json address: 0.0.0.0:4317 rate_limit: "" ``` #### Advanced ```yml input: label: "" otlp_grpc: encoding: json address: 0.0.0.0:4317 tls: enabled: false cert_file: "" key_file: "" auth_token: "" max_recv_msg_size: 4194304 rate_limit: "" tcp: reuse_addr: false reuse_port: false schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} common_subject: "" trace_subject: "" log_subject: "" metric_subject: "" ``` ## [](#protocols)Protocols This input supports OTLP/gRPC on the default port 4317 using the standard OTLP protobuf format for all signal types (traces, logs, metrics). ## [](#output-format)Output format Each OTLP export request is unbatched into individual messages: - **Traces**: One message per span - **Logs**: One message per log record - **Metrics**: One message per metric Messages are encoded in Redpanda OTEL v1 protobuf format. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `signal_type` - The signal type: "trace", "log", or "metric" You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication When `auth_token` is configured, clients must include the token in the gRPC metadata. ### [](#go-client-example)Go client example ```go import ( "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" ) exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint("localhost:4317"), otlptracegrpc.WithInsecure(), // or WithTLSCredentials() for TLS otlptracegrpc.WithHeaders(map[string]string{ "authorization": "Bearer your-token-here", }), ) ``` ### [](#environment-variable)Environment variable ```bash export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer your-token-here" ``` ## [](#rate-limiting)Rate limiting An optional rate limit resource can be specified to throttle incoming requests. When the rate limit is breached, requests will receive a ResourceExhausted gRPC status code. ## [](#fields)Fields ### [](#address)`address` The address to listen on for gRPC connections. **Type**: `string` **Default**: `0.0.0.0:4317` ### [](#auth_token)`auth_token` Optional bearer token for authentication. When set, requests must include 'authorization: Bearer ' metadata. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#encoding)`encoding` The encoding format for messages in each batch, either `protobuf` or `json`. Requires version 4.79.0 or later. **Type**: `string` **Default**: `json` **Options**: `protobuf`, `json` ### [](#max_recv_msg_size)`max_recv_msg_size` Maximum size of gRPC messages to receive in bytes. **Type**: `int` **Default**: `4194304` ### [](#rate_limit)`rate_limit` The name of an optional rate limit resource to throttle incoming requests. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Optional Schema Registry configuration. When set, the input registers the OTLP schemas for each signal type and prepends the Schema Registry wire format header to each message. Requires version 4.79.0 or later. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-common_subject)`schema_registry.common_subject` Schema subject name for the common protobuf schema. Only used when encoding is 'protobuf'. Defaults to 'redpanda-otel-common' for protobuf encoding or 'redpanda-otel-common-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-log_subject)`schema_registry.log_subject` Schema subject name for log data. Defaults to 'redpanda-otel-logs' for protobuf encoding or 'redpanda-otel-logs-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-metric_subject)`schema_registry.metric_subject` Schema subject name for metric data. Defaults to 'redpanda-otel-metrics' for protobuf encoding or 'redpanda-otel-metrics-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. Requires version 4.82.0 or later. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for Schema Registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-trace_subject)`schema_registry.trace_subject` Schema subject name for trace data. Defaults to 'redpanda-otel-traces' for protobuf encoding or 'redpanda-otel-traces-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-url)`schema_registry.url` Schema Registry URL for schema operations. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS configuration for gRPC. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file. **Type**: `string` **Default**: `""` --- # Page 74: otlp_http **URL**: https://docs.redpanda.com/connect/components/inputs/otlp_http.md --- # otlp_http --- title: otlp_http latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/otlp_http page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/otlp_http.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/otlp_http.adoc description: Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Introduced in version 4.78.0. Exposes an OpenTelemetry Collector HTTP receiver that accepts traces, logs, and metrics via HTTP. Telemetry data is received in OTLP format (both protobuf and JSON) at standard OTLP endpoints and converted to individual Redpanda OTEL v1 protobuf messages. Each signal (span, log record, or metric) becomes a separate message with embedded Resource and Scope metadata, optimized for Kafka partitioning. #### Common ```yml input: label: "" otlp_http: encoding: json address: 0.0.0.0:4318 rate_limit: "" ``` #### Advanced ```yml input: label: "" otlp_http: encoding: json address: 0.0.0.0:4318 tls: enabled: false cert_file: "" key_file: "" auth_token: "" read_timeout: 10s write_timeout: 10s max_body_size: 4194304 rate_limit: "" tcp: reuse_addr: false reuse_port: false schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} common_subject: "" trace_subject: "" log_subject: "" metric_subject: "" ``` ## [](#endpoints)Endpoints This input exposes the following standard OTLP HTTP endpoints: - `/v1/traces` - OpenTelemetry traces - `/v1/logs` - OpenTelemetry logs - `/v1/metrics` - OpenTelemetry metrics ## [](#protocols)Protocols This input supports OTLP/HTTP on the default port 4318. It accepts both: - `application/x-protobuf` - OTLP protobuf format - `application/json` - OTLP JSON format ## [](#output-format)Output format Each OTLP export request is unbatched into individual messages: - **Traces**: One message per span - **Logs**: One message per log record - **Metrics**: One message per metric Messages are encoded in Redpanda OTEL v1 protobuf format. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `signal_type` - The signal type: "trace", "log", or "metric" You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication When `auth_token` is configured, clients must include the token in the HTTP Authorization header. ### [](#go-client-example)Go client example ```go import ( "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" ) exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint("localhost:4318"), otlptracehttp.WithInsecure(), // or WithTLSClientConfig() for TLS otlptracehttp.WithHeaders(map[string]string{ "Authorization": "Bearer your-token-here", }), ) ``` ### [](#curl-example)cURL example ```bash curl -X POST http://localhost:4318/v1/traces \ -H "Content-Type: application/x-protobuf" \ -H "Authorization: Bearer your-token-here" \ --data-binary @traces.pb ``` ### [](#environment-variable)Environment variable ```bash export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-token-here" ``` ## [](#rate-limiting)Rate limiting An optional rate limit resource can be specified to throttle incoming requests. When the rate limit is breached, requests will receive a 429 (Too Many Requests) response. ## [](#fields)Fields ### [](#address)`address` The address to listen on for HTTP connections. **Type**: `string` **Default**: `0.0.0.0:4318` ### [](#auth_token)`auth_token` Optional bearer token for authentication. When set, requests must include 'Authorization: Bearer ' header. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#encoding)`encoding` The encoding format for messages in each batch, either `protobuf` or `json`. Requires version 4.79.0 or later. **Type**: `string` **Default**: `json` **Options**: `protobuf`, `json` ### [](#max_body_size)`max_body_size` Maximum size of HTTP request body in bytes. **Type**: `int` **Default**: `4194304` ### [](#rate_limit)`rate_limit` The name of an optional rate limit resource to throttle incoming requests. **Type**: `string` **Default**: `""` ### [](#read_timeout)`read_timeout` Maximum duration for reading the entire request. **Type**: `string` **Default**: `10s` ### [](#schema_registry)`schema_registry` Optional Schema Registry configuration. When set, the input registers the OTLP schemas for each signal type and prepends the Schema Registry wire format header to each message. Requires version 4.79.0 or later. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-common_subject)`schema_registry.common_subject` Schema subject name for the common protobuf schema. Only used when encoding is 'protobuf'. Defaults to 'redpanda-otel-common' for protobuf encoding or 'redpanda-otel-common-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-log_subject)`schema_registry.log_subject` Schema subject name for log data. Defaults to 'redpanda-otel-logs' for protobuf encoding or 'redpanda-otel-logs-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-metric_subject)`schema_registry.metric_subject` Schema subject name for metric data. Defaults to 'redpanda-otel-metrics' for protobuf encoding or 'redpanda-otel-metrics-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. Requires version 4.82.0 or later. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for Schema Registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-trace_subject)`schema_registry.trace_subject` Schema subject name for trace data. Defaults to 'redpanda-otel-traces' for protobuf encoding or 'redpanda-otel-traces-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-url)`schema_registry.url` Schema Registry URL for schema operations. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS configuration for HTTP. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file. **Type**: `string` **Default**: `""` ### [](#write_timeout)`write_timeout` Maximum duration for writing the response. **Type**: `string` **Default**: `10s` --- # Page 75: parquet **URL**: https://docs.redpanda.com/connect/components/inputs/parquet.md --- # parquet --- title: parquet latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/parquet page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/parquet.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/parquet.adoc description: Reads and decodes Parquet files into a stream of structured messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads and decodes [Parquet files](https://parquet.apache.org/docs/) into a stream of structured messages. Introduced in version 4.8.0. #### Common ```yml input: label: "" parquet: paths: [] # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" parquet: paths: [] # No default (required) batch_count: 1 auto_replay_nacks: true ``` This input uses [https://github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go), which is itself experimental. Therefore changes could be made into how this processor functions outside of major version releases. By default any BYTE\_ARRAY or FIXED\_LEN\_BYTE\_ARRAY value will be extracted as a byte slice (`[]byte`) unless the logical type is UTF8, in which case they are extracted as a string (`string`). When a value extracted as a byte slice exists within a document which is later JSON serialized by default it will be base 64 encoded into strings, which is the default for arbitrary data fields. It is possible to convert these binary values to strings (or other data types) using Bloblang transformations such as `root.foo = this.foo.string()` or `root.foo = this.foo.encode("hex")`, etc. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` Optionally process records in batches. This can help to speed up the consumption of exceptionally large files. When the end of the file is reached the remaining records are processed as a (potentially smaller) batch. **Type**: `int` **Default**: `1` ### [](#paths)`paths[]` A list of file paths to read from. Each file will be read sequentially until the list is exhausted, at which point the input will close. Glob patterns are supported, including super globs (double star). **Type**: `array` ```yaml # Examples: paths: /tmp/foo.parquet # --- paths: /tmp/bar/*.parquet # --- paths: /tmp/data/**/*.parquet ``` --- # Page 76: postgres_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/postgres_cdc.md --- # postgres_cdc --- title: postgres_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/postgres_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/postgres_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/postgres_cdc.adoc description: Streams changes from a PostgreSQL database using logical replication. page-git-created-date: "2024-12-05" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data changes from a PostgreSQL database using logical replication. There is also a configuration option to [stream all existing data](#stream_snapshot) from the database. Introduced in version 4.40.0 and renamed from `pg_stream` to `postgres_cdc` in version 4.43.0. ### Common ```yml input: label: "" postgres_cdc: dsn: "" # No default (required) include_transaction_markers: false stream_snapshot: false snapshot_batch_size: 1000 schema: "" # No default (required) tables: [] # No default (required) checkpoint_limit: 1024 temporary_slot: false slot_name: "" # No default (required) pg_standby_timeout: 10s pg_wal_monitor_interval: 3s max_parallel_snapshot_tables: 1 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml input: label: "" postgres_cdc: dsn: "" # No default (required) include_transaction_markers: false stream_snapshot: false snapshot_batch_size: 1000 schema: "" # No default (required) tables: [] # No default (required) checkpoint_limit: 1024 temporary_slot: false slot_name: "" # No default (required) pg_standby_timeout: 10s pg_wal_monitor_interval: 3s max_parallel_snapshot_tables: 1 unchanged_toast_value: null heartbeat_interval: 1h tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (required) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) signal_table_name: "" incremental_snapshot: enabled: false chunk_size: 1024 heartbeat_interval: 1s retry_cooldown: 30s checkpoint_cache: "" # No default (optional) checkpoint_cache_key: postgres_cdc_incremental_snapshot auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` The `postgres_cdc` input uses logical replication to capture changes made to a PostgreSQL database in real time and streams them to Redpanda Connect. Redpanda Connect uses this replication method to allow you to choose which database tables in your source database to receive changes from. There are also [two replication modes](#choose-a-replication-mode) to choose from, and an [option to receive TOAST and deleted values](#receive-toast-and-deleted-values) in your data updates. ## [](#prerequisites)Prerequisites - PostgreSQL version 14 or later - Logical replication enabled on your PostgreSQL cluster To check whether logical replication is already enabled, run the following query: ```SQL SHOW wal_level; ``` If the `wal_level` value is `logical`, you can start to use this connector. Otherwise, choose from the following sets of instructions to update your replication settings. ### Cloud platforms - [Amazon RDS for PostgreSQL DB](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication.html) - [Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical#prerequisites-for-logical-replication-and-logical-decoding) - [Google Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication), including creating a user with replication privileges - [Neon](https://neon.tech/docs/guides/logical-replication-guide) ### Self-Hosted PostgreSQL Use an account with sufficient permissions (superuser) to update your replication settings. 1. Open the `postgresql.conf` file. 2. Find the `wal_level` parameter. 3. Update the parameter value to `wal_level = logical`. If you already use replication slots, you may need to increase the limit on replication slots (`max_replication_slots`). The `max_wal_senders` parameter value must also be greater than or equal to `max_replication_slots`. 4. Restart the PostgreSQL server. For this input to make a successful connection to your database, also make sure that it allows replication connections. 1. Open the `pg_hba.conf` file. 2. Update this line. ```yaml host replication /32 md5 ``` Replace the following placeholders with your own values: - ``: The username from an account with superuser privileges. - ``: The IP address of the server where you are running Redpanda Connect. 3. Restart the PostgreSQL server. ## [](#choose-a-replication-mode)Choose a replication mode When you run a pipeline that uses the `postgres_cdc` input, Redpanda Connect connects to your PostgreSQL database and creates a replication slot. The replication slot uses a copy of the Write-Ahead Log (WAL) file to subscribe to changes in your database records as they are applied to the database. There are two replication modes you can choose from: snapshot mode and streaming mode. In snapshot mode, Redpanda Connect first takes a snapshot of the database and streams the contents before processing changes from the WAL. In streaming mode, Redpanda Connect directly processes changes from the WAL starting from the most recent changes without taking a snapshot first. For local testing, you can use the [example pipeline on this page](#example-pipeline), which runs in snapshot mode. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect: 1. Creates a snapshot of your database. 2. Streams the contents of the tables specified in the `postgres_cdc` input. 3. Starts processing changes in the WAL that occurred since the snapshot was taken, and streams them to Redpanda Connect. Once the initial replication process is complete, the snapshot is removed and the input keeps a connection open to the database so that it can receive data updates. If the pipeline restarts during the replication process, Redpanda Connect resumes processing data changes from where it left off. If there are other interruptions while the snapshot is taken, you may need to restart the snapshot process. For more information, see [Troubleshoot replication failures](#troubleshoot-replication-failures). ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect starts processing data changes from the end of the WAL. If the pipeline restarts, Redpanda Connect resumes processing data changes from the last acknowledged position in the WAL. ## [](#monitor-the-replication-process)Monitor the replication process You can monitor the initial replication of data using the following metrics: | Metric name | Description | | --- | --- | | replication_lag_bytes | Indicates how far the connector is lagging behind the source database when processing the transaction log. | | postgres_snapshot_progress | Shows the progress of snapshot processing for each table. | ## [](#troubleshoot-replication-failures)Troubleshoot replication failures If the database snapshot fails, the replication slot has only an incomplete record of the existing data in your database. To maintain data integrity, you must drop the replication slot manually in your source database and run the Redpanda Connect pipeline again. ```SQL SELECT pg_drop_replication_slot(SLOT_NAME); ``` ## [](#receive-toast-and-deleted-values)Receive TOAST and deleted values For full visibility of all data updates, you can also choose to stream [TOAST](https://www.postgresql.org/docs/current/storage-toast.html) and deleted values. To enable this option, run the following query on your source database: ```SQL ALTER TABLE large_data REPLICA IDENTITY FULL; ``` ## [](#data-mappings)Data mappings The following table shows how selected PostgreSQL data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values. | PostgreSQL data type | Bloblang value | | --- | --- | | TEXT, TIMESTAMP, UUID, VARCHAR | JSON strings, for example: this data | | BOOL | Boolean JSON fields, for example: true or false | | Numeric types (INT4) | JSON number types, for example: 1. | | JSONB | JSON objects, for example: { "message": "message text" } | | INTEGER[] | An array of integer values, for example: [1,2,3] | | TEXT[] | An array of string values, for example: ["value1", "value2", "value3"] | | INET | A string that contains an IP address, for example: "192.168.1.1" | | POINT | A string that represents a point in a two-dimensional plane, for example: (x, y) | | TSRANGE | A string that includes range bounds, for example: [2010-01-01 14:30, 2010-01-01 15:30) | | TSVECTOR | A string that includes vector data, for example: "'the':2 'question':3 'is':4" | ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `table`: Name of the table that the message originated from - `operation`: Type of operation that generated the message: "read", "insert", "update", or "delete". "read" is from messages that are read in the initial snapshot phase. This will also be "begin" and "commit" if `include_transaction_markers` is enabled - `lsn`: the log sequence number in postgres - `schema`: The table schema in benthos common schema format, compatible with processors like parquet\_encode - `commit_ts_ms`: The commit timestamp of the transaction as a Unix millisecond timestamp. Not set for snapshot reads. - `before`: The pre-change state of the row for update and delete operations, in benthos common schema format. For updates, availability depends on the table’s REPLICA IDENTITY setting - with the default identity only key columns are present, with REPLICA IDENTITY FULL all columns are present. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication configuration for PostgreSQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. This is useful for connecting to Amazon RDS or Aurora PostgreSQL instances with IAM database authentication enabled. The generated tokens are valid for 15 minutes and are automatically refreshed. For more information about AWS credentials configuration, see the [credentials for AWS](https://docs.redpanda.com/connect/guides/cloud/aws/) guide. Requires version 4.71.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication for PostgreSQL. When enabled, an IAM authentication token is generated and used as the password. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` The PostgreSQL endpoint hostname (for example, mydb.abc123.us-east-1.rds.amazonaws.com). **Type**: `string` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region where the PostgreSQL instance is located. The region is used to sign the IAM authentication token and for STS calls when assuming roles. If no region is specified, the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. When `roles` is also set, this role is assumed first and the `roles` entries are assumed after it. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. When `role` is also set, it is assumed before the first entry. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given log sequence number (LSN) is not acknowledged until all messages up to that position are delivered. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` The data source name (DSN) of the PostgreSQL database from which you want to stream updates. Use the format `postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​]`. PostgreSQL enforces SSL by default. To disable SSL, for example in a secure environment, add `sslmode=disable` to the connection string. **Type**: `string` ```yaml # Examples: dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable ``` ### [](#heartbeat_interval)`heartbeat_interval` The interval between heartbeat messages, which Redpanda Connect writes to the write-ahead log (WAL) using the `pg_logical_emit_message` function. Heartbeat messages are useful when you subscribe to data changes from tables with low activity, while other tables in the database have higher-frequency updates. Without new messages to acknowledge, PostgreSQL cannot reclaim the WAL, which can exhaust the local disk. Heartbeat messages allow Redpanda Connect to periodically acknowledge new messages even when no data updates occur. Each acknowledgement advances the committed point in the WAL, which ensures that PostgreSQL can safely reclaim older log segments. Set `heartbeat_interval` to `0s` to disable heartbeats. Heartbeats also pace `incremental_snapshot`. On a quiet table they’re the only thing that advances the snapshot. This interval keeps the slot current, and `incremental_snapshot.heartbeat_interval` applies alongside it: the more frequent of the two wins for the life of the input. A non-zero value is still required here. Requires version 4.48.0 or later. **Type**: `string` **Default**: `1h` ```yaml # Examples: heartbeat_interval: 0s # --- heartbeat_interval: 24h ``` ### [](#include_transaction_markers)`include_transaction_markers` When set to `true`, creates empty messages for the `BEGIN` and `COMMIT` operations that start and complete each transaction. Messages with the `operation` metadata field set to `begin` or `commit` have null message payloads. **Type**: `bool` **Default**: `false` ### [](#incremental_snapshot)`incremental_snapshot` Configures chunked snapshotting that runs alongside replication streaming. **Type**: `object` ### [](#incremental_snapshot-checkpoint_cache)`incremental_snapshot.checkpoint_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) storing the snapshot’s progress, so a restart resumes instead of starting over. Required when `enabled` is `true`. **Type**: `string` ### [](#incremental_snapshot-checkpoint_cache_key)`incremental_snapshot.checkpoint_cache_key` The key used to store the incremental snapshot progress in `checkpoint_cache`. Use a different key if multiple incremental snapshots share the same cache. Changing or clearing this key discards the record of which tables have been backfilled, so a `snapshot-execute` signal reads a table again. **Type**: `string` **Default**: `postgres_cdc_incremental_snapshot` ### [](#incremental_snapshot-chunk_size)`incremental_snapshot.chunk_size` The number of rows to read per chunk while incrementally snapshotting a table. **Type**: `int` **Default**: `1024` ### [](#incremental_snapshot-enabled)`incremental_snapshot.enabled` Backfills tables in chunks alongside replication, on request. Tables are not configured here: insert a `snapshot-execute` row into `signal_table_name` to ask for one, so a backfill can be started at any time without a config change. A signal table is therefore required. Unlike `stream_snapshot` this needs no up-front snapshot phase and does not delay replication. The two are mutually exclusive: both read the same rows, so enabling either alongside the other would deliver everything twice. Progress is driven by the replication stream: each streamed transaction releases a buffered chunk, and several more follow immediately if the database was idle during the read. Quiet tables therefore advance in bursts on each heartbeat, paced by `heartbeat_interval`. A row can arrive twice, once from replication and once from the backfill: when a primary key reuses or fills a gap below the table’s current maximum, or, whatever the key type, when a row is inserted after replication starts but before the snapshot reaches its table. Treat rows as idempotent upserts keyed by primary key, as is standard CDC practice. The following primary keys are currently not supported and a signal naming such a table is rejected and logged rather than started: `bytea`, `interval`, `bit`, `bit varying`, and any range or multirange type. Every other key type is supported, composite keys included. **Type**: `bool` **Default**: `false` ### [](#incremental_snapshot-heartbeat_interval)`incremental_snapshot.heartbeat_interval` How often to heartbeat while `enabled` is `true`. The snapshot only advances on a streamed transaction, so on quiet tables this paces it. Raise it to reduce write load at the cost of a slower backfill. Whichever of this and the top-level `heartbeat_interval` is more frequent wins, and applies for the life of the input: it is fixed at startup, and stays in force between backfills as well as during them. Heartbeats are transactional only while a backfill is in progress, since that is the only time the snapshot needs a transaction id from one; between backfills they cost nothing extra. **Type**: `string` **Default**: `1s` ### [](#incremental_snapshot-retry_cooldown)`incremental_snapshot.retry_cooldown` How long the backfill waits before retrying a chunk read that failed transiently, usually because of a conflicting lock on the table being backfilled, such as `VACUUM FULL` or most `ALTER TABLE` statements. The snapshot’s chunk read shares the goroutine that reads the replication stream, so each retry that blocks on the lock delays replication for every table. Retrying on every streamed transaction would repeat that delay for as long as the lock is held. Raise it to protect replication latency during a long migration, or lower it to resume the backfill sooner after a brief lock. Setting it to `0s` disables the cooldown, retrying on the next streamed transaction. **Type**: `string` **Default**: `30s` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specify the maximum number of tables that are processed in parallel when the initial snapshot of the source database is taken. **Type**: `int` **Default**: `1` ### [](#pg_standby_timeout)`pg_standby_timeout` Specify the standby timeout after which an idle connection is refreshed to keep the connection alive. **Type**: `string` **Default**: `10s` ```yaml # Examples: pg_standby_timeout: 30s ``` ### [](#pg_wal_monitor_interval)`pg_wal_monitor_interval` How often to report changes to the replication lag and write them to Redpanda Connect metrics. **Type**: `string` **Default**: `3s` ```yaml # Examples: pg_wal_monitor_interval: 6s ``` ### [](#schema)`schema` The PostgreSQL schema from which to replicate data. **Type**: `string` ```yaml # Examples: schema: public # --- schema: '"MyCaseSensitiveSchemaNeedingQuotes"' ``` ### [](#signal_table_name)`signal_table_name` The name of the table used to send control signals to the connector, excluding the schema. The table must exist in the schema configured via the `schema` field, and must not also appear in `tables`, since the signal table is implicitly added to the publication and excluded from snapshot scans, so listing it in both places is rejected at startup. It must have at least these columns; startup validation checks column names only, not types, so a wrong column type (for example `data JSONB` instead of `TEXT`) is only caught at runtime, on the first signal row read: - **id**: any type representable as a string (for example `SERIAL`, `BIGSERIAL`, `UUID`, `VARCHAR`) - **type**: should be `VARCHAR` or another string type (the signal type, see supported signals below) - **data**: should be `TEXT`, a JSON object containing signal parameters Create the table with: ```sql CREATE TABLE . ( id SERIAL PRIMARY KEY, type VARCHAR(32), data TEXT ); ``` Signal rows are published as regular output messages (`operation=insert`, `table=`). To exclude them from downstream processing, filter on the `table` metadata field using a `mapping` processor: ```yaml pipeline: processors: - mapping: | root = if @table == "rpcn_signal_table" { deleted() } else { this } ``` **Supported signals** **`log`**: recognized and logged when received. The `data` column must contain a JSON object with a `message` key, whose value is written to the connector’s log output. ```sql INSERT INTO . (type, data) VALUES ('log', '{"message": "Signal message"}'); ``` **`snapshot-execute`**: backfills the named tables incrementally, alongside streaming. Requires `incremental_snapshot.enabled`. The `data` column must contain a JSON object with a `tables` key listing table names in the configured `schema`, excluding the schema itself: ```sql INSERT INTO . (type, data) VALUES ('snapshot-execute', '{"tables": ["orders", "customers"]}'); ``` Each table must appear in `tables` (or that list must be empty, replicating everything): an unreplicated table has no live changes to deduplicate its backfill against, so a write landing after its chunk is read would be lost. Each must also have a primary key, which the backfill pages by. A table replicated under `REPLICA IDENTITY FULL` without one cannot be snapshotted. That key may not be `bytea`: its value is read back and bound as the next chunk’s bound, and raw bytes survive neither that nor the checkpoint. **Partitioned tables** Partitioned tables are not yet supported in incremental snapshotting unless their publication sets `publish_via_partition_root = true`. PostgreSQL otherwise publishes their changes under the individual partitions' names while the snapshot backfill reads the parent, so updates to already-read snapshot rows cannot be detected. A signal naming a table with partitions is rejected and logged, and it is left to replication alone. If a check cannot be run at all (for example, because the connection resets), the stream restarts and the signal is read again, so the request is not lost. Each table joins the back of the backfill queue. A table this run already covers is skipped and logged, so a repeated signal does not re-read it. To read one again, point `incremental_snapshot.checkpoint_cache_key` at a fresh key. **Tables with large (TOASTed) column values** Set `REPLICA IDENTITY FULL` on a table with large (TOASTed) column values before backfilling using incremental snapshotting. PostgreSQL omits an unchanged TOAST value from an `UPDATE`, sending a marker instead, and under the default replica identity there is nothing in the message to recover it from, so `unchanged_toast_value` is emitted for that column. For a row updated while its chunk is buffered, the backfilled copy that held the real value is dropped as a duplicate, leaving the placeholder as the only value the destination ever receives for it. `REPLICA IDENTITY FULL` makes PostgreSQL send the previous row, which the connector reads the real value from. It can be set for the backfill and reverted afterwards; it takes a brief lock but rewrites nothing. Requires version 4.105.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: signal_table_name: rpcn_signal_table ``` ### [](#slot_name)`slot_name` The name of the PostgreSQL logical replication slot to use. If the slot does not exist, the input creates it. You can also create the slot manually before starting replication. To avoid granting the replication user permission to create publications, you can create the publications manually ahead of time. This input uses the naming pattern `pglog_stream_`, so create publications using this convention. Starting with version 4.48.0, this input no longer adds the prefix `rs_` to the names of the replication slots it creates. To keep using a replication slot that an earlier version created, add the `rs_` prefix to this field yourself. **Type**: `string` ```yaml # Examples: slot_name: my_test_slot ``` ### [](#snapshot_batch_size)`snapshot_batch_size` The number of table rows to fetch in each batch when querying the snapshot. This option is only available when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1000` ```yaml # Examples: snapshot_batch_size: 10000 ``` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source database before streaming data changes. To use this setting, all database tables that you want to replicate _must_ have a primary key, which allows the input to read each table in parallel. This setting has no effect if `tables` is left empty, since the snapshot is only planned for tables listed there. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` ### [](#tables)`tables[]` A list of database table names to include in the snapshot and logical replication. Specify each table name as a separate item. If left empty, the underlying PostgreSQL publication is created `FOR ALL TABLES`, which replicates every table in every schema of the database, ignoring `schema`. This also disables `stream_snapshot`, since the initial snapshot is only planned for tables listed here. **Type**: `array` ```yaml # Examples: tables: - my_table_1 - '"MyCaseSensitiveTableNeedingQuotes"' ``` ### [](#temporary_slot)`temporary_slot` If set to `true`, the input creates a temporary replication slot that is automatically dropped when the connection to your source database is closed. You might use this option to: - Avoid data accumulating in the replication slot when a pipeline is paused or stopped. - Test the connector. If the pipeline is restarted and `stream_snapshot` is enabled, another data snapshot is taken before data updates are streamed. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings for the PostgreSQL connection. When `enabled` is `true`, these settings replace the TLS settings derived from the `dsn` and from `PG*` environment variables such as `PGSSLMODE`, and the server name is set to the host from the DSN. Requires version 4.70.0 or later. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#unchanged_toast_value)`unchanged_toast_value` Specify the value to emit when unchanged TOAST values appear in the message stream. Unchanged values occur for data updates and deletes when `REPLICA IDENTITY` is not set to `FULL`. Prefer a distinctive sentinel over the `null` default. A `null` can’t be told apart from a column that is genuinely null, so a consumer can’t skip the field instead of overwriting a good value with it. This matters most alongside `incremental_snapshot`, where a backfilled row can be the only delivery that carries a large column’s real value. See [`signal_table_name`](#signal_table_name). Requires version 4.46.0 or later. **Type**: `unknown` **Default**: `null` ```yaml # Examples: unchanged_toast_value: __redpanda_connect_unchanged_toast_value__ ``` ## [](#example-pipeline)Example pipeline You can run the following pipeline locally to check that data updates are streamed from your source database to Redpanda Connect. All transactions are written to stdout. ```yml input: label: "postgres_cdc" postgres_cdc: dsn: postgres://user:password@host:port/dbname include_transaction_markers: false slot_name: test_slot_native_decoder snapshot_batch_size: 100000 stream_snapshot: true temporary_slot: true schema: schema_name tables: - table_name cache_resources: - label: data_caching file: directory: /tmp/cache output: label: main stdout: {} ``` --- # Page 77: pulsar **URL**: https://docs.redpanda.com/connect/components/inputs/pulsar.md --- # pulsar --- title: pulsar latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/pulsar page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/pulsar.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/pulsar.adoc description: Reads messages from an Apache Pulsar server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads messages from an Apache Pulsar server. Introduced in version 3.43.0. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `pulsar_message_id` - `pulsar_key` - `pulsar_ordering_key` - `pulsar_event_time_unix` - `pulsar_publish_time_unix` - `pulsar_topic` - `pulsar_producer_name` - `pulsar_redelivery_count` - All properties of the message You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" pulsar: url: "" # No default (required) topics: [] # No default (optional) topics_pattern: "" # No default (optional) subscription_name: "" # No default (required) subscription_type: shared subscription_initial_position: latest tls: root_cas_file: "" ``` ### Advanced ```yml input: label: "" pulsar: url: "" # No default (required) topics: [] # No default (optional) topics_pattern: "" # No default (optional) subscription_name: "" # No default (required) subscription_type: shared subscription_initial_position: latest tls: root_cas_file: "" auth: oauth2: enabled: false audience: "" issuer_url: "" scope: "" private_key_file: "" token: enabled: false token: "" ``` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of Pulsar authentication methods. Requires version 3.60.0 or later. **Type**: `object` ### [](#auth-oauth2)`auth.oauth2` Parameters for Pulsar OAuth2 authentication. **Type**: `object` ### [](#auth-oauth2-audience)`auth.oauth2.audience` OAuth2 audience. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-enabled)`auth.oauth2.enabled` Whether OAuth2 is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-oauth2-issuer_url)`auth.oauth2.issuer_url` OAuth2 issuer URL. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-private_key_file)`auth.oauth2.private_key_file` The path to a file containing a private key. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-scope)`auth.oauth2.scope` OAuth2 scope to request. Requires version 4.29.0 or later. **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Parameters for Pulsar Token authentication. **Type**: `object` ### [](#auth-token-enabled)`auth.token.enabled` Whether Token Auth is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-token-token)`auth.token.token` Actual base64 encoded token. **Type**: `string` **Default**: `""` ### [](#subscription_initial_position)`subscription_initial_position` Specify the subscription initial position for this consumer. Requires version 4.29.0 or later. **Type**: `string` **Default**: `latest` **Options**: `latest`, `earliest` ### [](#subscription_name)`subscription_name` Specify the subscription name for this consumer. **Type**: `string` ### [](#subscription_type)`subscription_type` Specify the subscription type for this consumer. > > 📝 **NOTE** > > > > Using a `key_shared` subscription type will _allow out-of-order delivery_ since nack-ing messages sets non-zero nack delivery delay - this can potentially cause consumers to stall. See [Pulsar documentation](https://pulsar.apache.org/docs/concepts-messaging/#negative-acknowledgment) and [this Github issue](https://github.com/apache/pulsar/issues/12208) for more details. **Type**: `string` **Default**: `shared` **Options**: `shared`, `key_shared`, `failover`, `exclusive` ### [](#tls)`tls` Specify the path to a custom CA certificate to trust the broker TLS service. Requires version 4.2.0 or later. **Type**: `object` ### [](#tls-root_cas_file)`tls.root_cas_file` The path of a PEM file containing the certificate authority certificates that the client trusts when it verifies the TLS certificate of the broker. Leave empty to trust the certificate authorities of the host system. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#topics)`topics[]` A list of topics to subscribe to. This or topics\_pattern must be set. **Type**: `array` ### [](#topics_pattern)`topics_pattern` A regular expression matching the topics to subscribe to. This or topics must be set. Requires version 4.19.0 or later. **Type**: `string` ### [](#url)`url` A URL to connect to. **Type**: `string` ```yaml # Examples: url: pulsar://localhost:6650 # --- url: pulsar://pulsar.us-west.example.com:6650 # --- url: pulsar+ssl://pulsar.us-west.example.com:6651 ``` --- # Page 78: read_until **URL**: https://docs.redpanda.com/connect/components/inputs/read_until.md --- # read_until --- title: read_until latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/read_until page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/read_until.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/read_until.adoc description: Reads messages from a child input until a consumed message passes a Bloblang query, at which point the input closes. It is also possible to configure a timeout after which the input is closed if no new messages arrive in that period. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads messages from a child input until a consumed message passes a [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/), at which point the input closes. It is also possible to configure a timeout after which the input is closed if no new messages arrive in that period. Messages are read continuously while the query check returns false, when the query returns true the message that triggered the check is sent out and the input is closed. Use this to define inputs where the stream should end once a certain message appears. If the idle timeout is configured, the input will be closed if no new messages arrive after that period of time. Use this field if you want to empty out and close an input that doesn’t have a logical end. Sometimes inputs close themselves. For example, when the `file` input type reaches the end of a file it will shut down. By default this type will also shut down. If you wish for the input type to be restarted every time it shuts down until the query check is met then set `restart_input` to `true`. ## [](#metadata)Metadata A metadata key `benthos_read_until` containing the value `final` is added to the first part of the message that triggers the input to stop. ```yml input: label: "" read_until: input: {} # No default (required) check: "" # No default (optional) idle_timeout: "" # No default (optional) restart_input: false ``` ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that should return a boolean value indicating whether the input should now be closed. **Type**: `string` ```yaml # Examples: check: this.type == "foo" # --- check: counter() >= 100 ``` ### [](#idle_timeout)`idle_timeout` The maximum amount of time without receiving new messages after which the input is closed. Requires version 4.24.0 or later. **Type**: `string` ```yaml # Examples: idle_timeout: 5s ``` ### [](#input)`input` The child input to consume from. **Type**: `input` ### [](#restart_input)`restart_input` Whether the input should be reopened if it closes itself before the condition has resolved to true. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#consume-n-messages)Consume N Messages A common reason to use this input is to consume only N messages from an input and then stop. This can easily be done with the [`counter` function](https://docs.redpanda.com/connect/guides/bloblang/functions/#counter): ```yaml # Only read 100 messages, and then exit. input: read_until: check: counter() >= 100 input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup ``` ### [](#read-from-a-kafka-and-close-when-empty)Read from a kafka and close when empty A common reason to use this input is a job that consumes all messages and exits once its empty: ```yaml # Consumes all messages and exit when the last message was consumed 5s ago. input: read_until: idle_timeout: 5s input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup ``` --- # Page 79: redis_list **URL**: https://docs.redpanda.com/connect/components/inputs/redis_list.md --- # redis_list --- title: redis_list latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redis_list page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_list.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_list.adoc description: Pops messages from the beginning of a Redis list using the BLPop command. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Pops messages from the beginning of a Redis list using the BLPop command. #### Common ```yml input: label: "" redis_list: url: "" # No default (required) key: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" redis_list: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) auto_replay_nacks: true max_in_flight: 0 timeout: 5s command: blpop ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command used to pop elements from the Redis list Requires version 4.22.0 or later. **Type**: `string` **Default**: `blpop` **Options**: `blpop`, `brpop` ### [](#key)`key` The key of a list to read from. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` Optionally sets a limit on the number of messages that can be flowing through a Redpanda Connect stream pending acknowledgment from the input at any given time. Once a message has been either acknowledged or rejected (nacked) it is no longer considered pending. If the input produces logical batches then each batch is considered a single count against the maximum. **WARNING**: Batching policies at the output level will stall if this field limits the number of messages below the batching threshold. Zero (default) or lower implies no limit. Requires version 4.9.0 or later. **Type**: `int` **Default**: `0` ### [](#timeout)`timeout` The length of time to poll for new messages before reattempting. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 80: redis_pubsub **URL**: https://docs.redpanda.com/connect/components/inputs/redis_pubsub.md --- # redis_pubsub --- title: redis_pubsub latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redis_pubsub page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_pubsub.adoc description: Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands. In order to subscribe to channels using the `PSUBSCRIBE` command set the field `use_patterns` to `true`, then you can include glob-style patterns in your channel names. For example: - `h?llo` subscribes to hello, hallo and hxllo - `h*llo` subscribes to hllo and heeeello - `h[ae]llo` subscribes to hello and hallo, but not hillo Use `\` to escape special characters if you want to match them verbatim. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `redis_pubsub_channel` - `redis_pubsub_pattern` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" redis_pubsub: url: "" # No default (required) channels: [] # No default (required) use_patterns: false auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" redis_pubsub: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] channels: [] # No default (required) use_patterns: false auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#channels)`channels[]` A list of channels to consume from. **Type**: `array` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#use_patterns)`use_patterns` Whether to use the PSUBSCRIBE command, allowing for glob-style patterns within target channel names. **Type**: `bool` **Default**: `false` --- # Page 81: redis_scan **URL**: https://docs.redpanda.com/connect/components/inputs/redis_scan.md --- # redis_scan --- title: redis_scan latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redis_scan page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_scan.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_scan.adoc description: Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands. Introduced in version 4.27.0. Optionally, iterates only elements matching a blob-style pattern. For example: - `**foo**` iterates only keys which contain `foo` in it. - `foo*` iterates only keys starting with `foo`. This input generates a message for each key value pair in the following format: ```json {"key":"foo","value":"bar"} ``` #### Common ```yml input: label: "" redis_scan: url: "" # No default (required) auto_replay_nacks: true match: "" ``` #### Advanced ```yml input: label: "" redis_scan: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true match: "" ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#match)`match` Iterates only elements matching the optional glob-style pattern. By default, it matches all elements. **Type**: `string` **Default**: `""` ```yaml # Examples: match: "*" # --- match: 1* # --- match: foo* # --- match: foo # --- match: "*4*" ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 82: redis_streams **URL**: https://docs.redpanda.com/connect/components/inputs/redis_streams.md --- # redis_streams --- title: redis_streams latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redis_streams page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_streams.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_streams.adoc description: Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The client_id should be unique for each consumer of a group. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The `client_id` should be unique for each consumer of a group. Redis stream entries are key/value pairs, as such it is necessary to specify the key that contains the body of the message. All other keys/value pairs are saved as metadata fields. #### Common ```yml input: label: "" redis_streams: url: "" # No default (required) body_key: body streams: [] # No default (required) auto_replay_nacks: true limit: 10 client_id: "" consumer_group: "" ``` #### Advanced ```yml input: label: "" redis_streams: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] body_key: body streams: [] # No default (required) auto_replay_nacks: true limit: 10 client_id: "" consumer_group: "" create_streams: true start_from_oldest: true commit_period: 1s timeout: 1s ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#body_key)`body_key` The field key to extract the raw message from. All other keys will be stored in the message as metadata. **Type**: `string` **Default**: `body` ### [](#client_id)`client_id` The consumer name used within `consumer_group` when reading with `XREADGROUP`. Redis tracks pending messages for each consumer name. **Type**: `string` **Default**: `""` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each batch of `XACK` calls that acknowledge delivered message IDs to the consumer group. Pending acknowledgements are also sent during shutdown. **Type**: `string` **Default**: `1s` ### [](#consumer_group)`consumer_group` An identifier for the consumer group of the stream. **Type**: `string` **Default**: `""` ### [](#create_streams)`create_streams` Create subscribed streams if they do not exist (MKSTREAM option). **Type**: `bool` **Default**: `true` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#limit)`limit` The maximum number of messages to consume from a single request. **Type**: `int` **Default**: `10` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#start_from_oldest)`start_from_oldest` If an offset is not found for a stream, determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. **Type**: `bool` **Default**: `true` ### [](#streams)`streams[]` A list of streams to consume from. **Type**: `array` ### [](#timeout)`timeout` The length of time to poll for new messages before reattempting. **Type**: `string` **Default**: `1s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 83: redpanda_common **URL**: https://docs.redpanda.com/connect/components/inputs/redpanda_common.md --- # redpanda_common --- title: redpanda_common latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redpanda_common page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redpanda_common.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redpanda_common.adoc description: Consumes data from a Redpanda (Kafka) broker, using credentials defined in a common top-level redpanda config block. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) components. Consumes data from a Redpanda (Kafka) broker, using credentials from a common `redpanda` configuration block. To avoid duplicating Redpanda cluster credentials in your `redpanda_common` input, output, or any other components in your data pipeline, you can use a single [`redpanda` configuration block](https://docs.redpanda.com/connect/components/redpanda/about/). For more details, see the [Pipeline example](#pipeline-example). Introduced in version 4.39.0. > 📝 **NOTE** > > If you need to move topic data between Redpanda clusters or other Apache Kafka clusters, consider using the [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [output](https://docs.redpanda.com/connect/components/outputs/redpanda/) instead. #### Common ```yml input: label: "" redpanda_common: topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" redpanda_common: topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#consumer-groups)Consumer groups When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. ### [](#delivery-guarantees)Delivery guarantees If you choose to use consumer groups, the offsets of records received by Redpanda Connect are committed automatically. In the event of restarts, this input uses the committed offsets to resume data consumption where it left off. Redpanda Connect guarantees at-least-once delivery. Records are only confirmed as delivered when all downstream outputs that a record is routed to have also confirmed delivery. ## [](#ordering)Ordering To preserve the order of topic partitions: - Records consumed from each partition are processed and delivered in the order that they are received - Only one batch of records of a given partition is processed at a time This approach means that although records from different partitions may be processed in parallel, records from the same partition are processed in sequential order. ### [](#delivery-errors)Delivery errors The order in which records are delivered may be disrupted by delivery errors and any error-handling mechanisms that start up. Redpanda Connect uses at-least-once delivery unless instructed otherwise, and this includes reattempting delivery of data when the ordering of that data is no longer guaranteed. For example, a batch of records is sent to an output broker and only a subset of records are delivered. In this scenario, Redpanda Connect (by default) attempts to deliver the records that failed, even though these delivery failures may have been sent before records that were delivered successfully. #### [](#use-a-fallback-output)Use a fallback output To prevent delivery errors from disrupting the order of records, you must specify a [`fallback`](https://docs.redpanda.com/connect/components/outputs/fallback/) output in your pipeline configuration. When adding a `fallback` output, it is good practice to set the `auto_retry_nacks` field to `false`. This also improves the throughput of your pipeline. For example, the following configuration includes a `fallback` output. If Redpanda Connect fails to write delivery errors to the `foo` topic, it then attempts to write them into a dead letter queue topic (`foo_dlq`), which is retried indefinitely as a way to apply back pressure. ```yaml output: fallback: - redpanda_common: topic: foo - retry: output: redpanda_common: topic: foo_dlq ``` ## [](#batching)Batching Records are processed and delivered from each partition in the same batches as they are received from brokers. Batch sizes are dynamically sized in order to optimize throughput, but you can tune them further using the following configuration fields: - `fetch_max_partition_bytes` - `fetch_max_bytes` You can break batches down further using the [`split`](https://docs.redpanda.com/connect/components/processors/split/) processor. ## [](#metrics)Metrics This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#consumer_group)`consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics specified in the `topics` field are automatically distributed across consumers that share the consumer group. - Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. This field is equivalent to the Java setting `fetch.max.bytes`. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). This field is equivalent to the Java setting `fetch.max.wait.ms`. Requires version 4.45.0 or later. **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. This value must be lower than `session_timeout`, and should be no higher than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a `consumer_group`, set `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. The value must be unique per consumer within the group. When you assign an instance ID, the client does not leave the consumer group when it closes. To remove the client from the group, you must use an external admin command on behalf of the instance ID. Requires version 4.46.0 or later. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. Requires version 4.55.0 or later. **Type**: `string` **Default**: `32KB` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a `consumer_group`, `rebalance_timeout` sets how long consumer group members can take to complete their work and commit offsets after a rebalance has begun. The time that a member takes to detect the rebalance (from a heartbeat) counts against this timeout. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. Requires version 4.69.0 or later. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. Requires version 4.69.0 or later. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. Requires version 4.53.0 or later. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.62.0 or later. **Type**: `string` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between consumer lag refreshes. During each cycle, this input asks the brokers for the consumer group’s committed offsets and the partition end offsets, and records the difference (the number of unread messages) for each topic partition in the consumer lag metric and the `kafka_lag` metadata field. Lag is only refreshed when `consumer_group` is set. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. You can list multiple comma-separated topics in a single element. If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusive. Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. Requires version 4.53.0 or later. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | --- # Page 84: redpanda_migrator **URL**: https://docs.redpanda.com/connect/components/inputs/redpanda_migrator.md --- # redpanda_migrator --- title: redpanda_migrator latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redpanda_migrator page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redpanda_migrator.adoc description: Kafka consumer for migration pipelines. All migration logic is handled by the redpanda_migrator output. page-git-created-date: "2024-10-02" page-git-modified-date: "2026-10-10" --- Unified Kafka consumer for migrating data between Kafka/Redpanda clusters. Use this input with the [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) to safely transfer topic data, ACLs, schemas, and consumer group offsets between clusters. This component is designed for migration scenarios. Introduced in version 4.67.5. ### Common ```yml input: label: "" redpanda_migrator: seed_brokers: [] # No default (required) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) schema_registry: url: "" # No default (required) timeout: 5s auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" redpanda_migrator: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} auto_replay_nacks: true ``` The `redpanda_migrator` input: - Reads a batch of messages from a broker. - Waits for the `redpanda_migrator` output to acknowledge the writes before updating the Kafka consumer group offset. - Provides the same delivery guarantees and ordering semantics as the [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/). Specify a consumer group to make this input consume one or more topics and automatically balance the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. This input requires a corresponding `redpanda_migrator` output in the same pipeline. Each pipeline must have both input and output components configured. For capabilities, guarantees, scheduling, and examples, see the output documentation. ## [](#requirements)Requirements - Must be paired with a `redpanda_migrator` output in the same pipeline. - Requires access to a source Kafka or Redpanda cluster. - Consumer group configuration is recommended for partition balancing. - When the source cluster enforces ACLs, a consumer ACL alone is not enough for the source principal: it needs at minimum topic `READ` **and** `DESCRIBE_CONFIGS`, plus consumer group and cluster permissions. A `READ` ACL grants `DESCRIBE` but not `DESCRIBE_CONFIGS`, so the migrator consumes messages but fails to create topics with `TOPIC_AUTHORIZATION_FAILED`. See [Required permissions](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/#required-permissions). ## [](#multiple-migrator-pairs)Multiple migrator pairs When using multiple migrator pairs in a single pipeline, coordination is based on the `label` field. The label of the input and output must match exactly for correct pairing. If labels do not match, migration fails for that pair. ## [](#performance-tuning-for-high-throughput)Performance tuning for high throughput For workloads with high message rates or large messages, adjust the following settings to optimize throughput: On this input component: - `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size - `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded On the paired `redpanda_migrator` output component: - `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster) > 📝 **NOTE** > > Setting `max_yield_batch_bytes` over 1MB is counter-productive unless you change the broker settings to allow bigger messages or batches. The `partition_buffer_bytes` setting allows for partition readahead. ## [](#metrics)Metrics This input emits an `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic. This metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. Monitor this metric to track migration progress and detect bottlenecks. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - kafka\_key - kafka\_topic - kafka\_partition - kafka\_offset - kafka\_lag - kafka\_timestamp\_ms - kafka\_timestamp\_unix - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics specified in the `topics` field are automatically distributed across consumers that share the consumer group. - Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. This field is equivalent to the Java setting `fetch.max.bytes`. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). This field is equivalent to the Java setting `fetch.max.wait.ms`. **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. This value must be lower than `session_timeout`, and should be no higher than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a `consumer_group`, set `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. The value must be unique per consumer within the group. When you assign an instance ID, the client does not leave the consumer group when it closes. To remove the client from the group, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a `consumer_group`, `rebalance_timeout` sets how long consumer group members can take to complete their work and commit offsets after a rebalance has begun. The time that a member takes to detect the rebalance (from a heartbeat) counts against this timeout. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. Requires version 4.69.0 or later. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. Requires version 4.69.0 or later. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for schema registry requests. Requires version 4.72.0 or later. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. Required for schema migration functionality. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 # --- url: https://schema-registry.example.com:8081 ``` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between consumer lag refreshes. During each cycle, this input asks the brokers for the consumer group’s committed offsets and the partition end offsets, and records the difference (the number of unread messages) for each topic partition in the consumer lag metric and the `kafka_lag` metadata field. Lag is only refreshed when `consumer_group` is set. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. You can list multiple comma-separated topics in a single element. If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusive. Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ## [](#troubleshooting)Troubleshooting - Ensure the input and output `label` fields match exactly. - Both input and output must be present in the pipeline. - Verify consumer group configuration for partition balancing. - Monitor the lag metric for stalled migration. ## [](#suggested-reading)Suggested reading - [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) - [Migrating from legacy components](https://docs.redpanda.com/connect/guides/migration/migrate-unified-redpanda-migrator/) --- # Page 85: redpanda **URL**: https://docs.redpanda.com/connect/components/inputs/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/redpanda page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redpanda.adoc description: A Kafka input using the Franz Kafka client library. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-14" --- Consumes topic data from one or more Kafka brokers. Introduced in version 4.39.0. To share one connection across several `redpanda` components, omit `seed_brokers` and configure the top-level [`redpanda` block](https://docs.redpanda.com/connect/components/redpanda/about/). #### Common ```yml input: label: "" redpanda: seed_brokers: [] # No default (optional) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" redpanda: seed_brokers: [] # No default (optional) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB unordered_processing: enabled: false checkpoint_limit: 1024 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#consumer-groups)Consumer groups When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. ## [](#delivery-guarantees)Delivery guarantees If you choose to use consumer groups, the offsets of records received by Redpanda Connect are committed automatically. In the event of restarts, this input uses the committed offsets to resume data consumption where it left off. Redpanda Connect guarantees at-least-once delivery. Records are only confirmed as delivered when all downstream outputs that a record is routed to have also confirmed delivery. ## [](#ordering)Ordering To preserve the order of topic partitions: - Records consumed from each partition are processed and delivered in the order that they are received - Only one batch of records of a given partition is processed at a time This approach means that although records from different partitions may be processed in parallel, records from the same partition are processed in sequential order. ### [](#delivery-errors)Delivery errors The order in which records are delivered may be disrupted by delivery errors and any error-handling mechanisms that start up. Redpanda Connect leans towards at-least-once delivery unless instructed otherwise, and this includes reattempting delivery of data when the ordering of that data is no longer guaranteed. For example, a batch of records is sent to an output broker and only a subset of records are delivered. In this scenario, Redpanda Connect (by default) attempts to deliver the records that failed, even though these delivery failures may have been sent before records that were delivered successfully. #### [](#use-a-fallback-output)Use a fallback output To prevent delivery errors from disrupting the order of records, you must specify a [`fallback`](https://docs.redpanda.com/connect/components/outputs/fallback/) output in your pipeline configuration. When adding a `fallback` output, it is good practice to set the `auto_retry_nacks` field to `false`. This also improves the throughput of your pipeline. For example, the following configuration includes a `fallback` output. If Redpanda Connect fails to write delivery errors to the `foo` topic, it then attempts to write them into a dead letter queue topic (`foo_dlq`), which is retried indefinitely as a way to apply back pressure. ```yaml output: fallback: - redpanda: seed_brokers: [ localhost:9092 ] topic: foo - retry: output: redpanda: seed_brokers: [ localhost:9092 ] topic: foo_dlq ``` ## [](#batching)Batching Records are processed and delivered from each partition in the same batches as they are received from brokers. Batch sizes are dynamically sized in order to optimize throughput, but you can tune them further using the following configuration fields: - `fetch_max_partition_bytes` - `fetch_max_bytes` You can break batches down further using the [`split`](https://docs.redpanda.com/connect/components/processors/split/) processor. ## [](#metrics)Metrics This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.53.0 or later. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics specified in the `topics` field are automatically distributed across consumers that share the consumer group. - Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.85.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. This field is equivalent to the Java setting `fetch.max.bytes`. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). This field is equivalent to the Java setting `fetch.max.wait.ms`. Requires version 4.45.0 or later. **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. This value must be lower than `session_timeout`, and should be no higher than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a `consumer_group`, set `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. The value must be unique per consumer within the group. When you assign an instance ID, the client does not leave the consumer group when it closes. To remove the client from the group, you must use an external admin command on behalf of the instance ID. Requires version 4.46.0 or later. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. Requires version 4.55.0 or later. **Type**: `string` **Default**: `32KB` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a `consumer_group`, `rebalance_timeout` sets how long consumer group members can take to complete their work and commit offsets after a rebalance has begun. The time that a member takes to detect the rebalance (from a heartbeat) counts against this timeout. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. Requires version 4.69.0 or later. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. Requires version 4.69.0 or later. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. Requires version 4.53.0 or later. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. If you omit this field, this component ignores the connection fields set on it, including `tls` and `sasl`, and takes its entire connection configuration from the top-level `redpanda` block. In Redpanda Cloud, the platform sets the top-level `redpanda` block for pipeline logs and status reporting, and its credentials can’t access your topics, so always set this field in Redpanda Cloud. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.46.0 or later. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. Requires version 4.53.0 or later. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.62.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between consumer lag refreshes. During each cycle, this input asks the brokers for the consumer group’s committed offsets and the partition end offsets, and records the difference (the number of unread messages) for each topic partition in the consumer lag metric and the `kafka_lag` metadata field. Lag is only refreshed when `consumer_group` is set. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. You can list multiple comma-separated topics in a single element. If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusive. Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. Requires version 4.53.0 or later. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ### [](#unordered_processing)`unordered_processing` Allows consumers to process messages of any given partition in parallel, which may result in unordered processing. This option enables asynchronous publishing at the output level. The maximum parallelization of each partition is determined by the `checkpoint_limit` field. Requires version 4.68.0 or later. **Type**: `object` ### [](#unordered_processing-batching)`unordered_processing.batching` Allows you to configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/) that applies to individual topic partitions in order to batch messages together before flushing them for processing. Batching can be beneficial for performance and useful for windowed processing, and doing so preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#unordered_processing-batching-byte_size)`unordered_processing.batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#unordered_processing-batching-check)`unordered_processing.batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#unordered_processing-batching-count)`unordered_processing.batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#unordered_processing-batching-period)`unordered_processing.batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#unordered_processing-batching-processors)`unordered_processing.batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#unordered_processing-checkpoint_limit)`unordered_processing.checkpoint_limit` The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value. **Type**: `int` **Default**: `1024` ### [](#unordered_processing-enabled)`unordered_processing.enabled` Whether to enable the unordered processing of messages from a given partition. **Type**: `bool` **Default**: `false` --- # Page 86: resource **URL**: https://docs.redpanda.com/connect/components/inputs/resource.md --- # resource --- title: resource latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/resource page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/resource.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/resource.adoc description: Resource is an input type that channels messages from a resource input, identified by its name. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Resource is an input type that channels messages from a resource input, identified by its name. ```yml input: resource: "" ``` Resources allow you to tidy up deeply nested configs. For example, the config: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ foo ] consumer_group: foogroup - gcp_pubsub: project: bar subscription: baz ``` Could also be expressed as: ```yaml input: broker: inputs: - resource: foo - resource: bar input_resources: - label: foo kafka: addresses: [ TODO ] topics: [ foo ] consumer_group: foogroup - label: bar gcp_pubsub: project: bar subscription: baz ``` Resources also allow you to reference a single input in multiple places, such as multiple streams mode configs, or multiple entries in a broker input. However, when a resource is referenced more than once the messages it produces are distributed across those references, so each message will only be directed to a single reference, not all of them. You can find out more about resources in [Resources](https://docs.redpanda.com/connect/configuration/resources/). --- # Page 87: salesforce_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/salesforce_cdc.md --- # salesforce_cdc --- title: salesforce_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/salesforce_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/salesforce_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/salesforce_cdc.adoc description: Subscribes to one or more Salesforce Pub/Sub topics in parallel and emits a message per event. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Subscribes to one or more Salesforce Pub/Sub topics in parallel and emits a message per event. Topics may be: - `/data/ChangeEvent`: per-sObject CDC channel. - `/data/ChangeEvents`: CDC firehose (every CDC-enabled sObject). - `/event/__e`: custom Platform Event. - `/event/`: standard Platform Event (for example `LoginEventStream`). - A bare sObject name (for example, `Account`) is shorthand for `/data/AccountChangeEvent`. Introduced in version 4.90.3. Optionally runs a REST snapshot for the CDC sObjects before opening the streaming subscriptions, so the pipeline sees the current state plus continuous changes. Per-topic replay state persists in a cache resource so each subscription resumes across restarts independently. ## [](#when-to-use-this-input)When to use this input Use `salesforce_cdc` for: - Continuous ingestion with both historical state (snapshot) and live changes (CDC). - Real-time custom or standard Platform Events. - Mixed CDC + Platform Event pipelines under a single component. Use a different Salesforce input instead if: - You only need a one-off extract or periodic SOQL query: use [`salesforce`](https://docs.redpanda.com/connect/components/inputs/salesforce/). - You need a GraphQL query (cross-object in one request): use [`salesforce_graphql`](https://docs.redpanda.com/connect/components/inputs/salesforce_graphql/). ### Common ```yml input: label: "" salesforce_cdc: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 topics: [] # No default (required) stream_snapshot: true replay_preset: latest snapshot_max_batch_size: 2000 stream_batch_size: 100 max_parallel_snapshot_objects: 1 checkpoint_cache: "" # No default (required) checkpoint_cache_key: salesforce_cdc checkpoint_limit: 1024 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml input: label: "" salesforce_cdc: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 topics: [] # No default (required) stream_snapshot: true replay_preset: latest snapshot_max_batch_size: 2000 stream_batch_size: 100 max_parallel_snapshot_objects: 1 checkpoint_cache: "" # No default (required) checkpoint_cache_key: salesforce_cdc checkpoint_limit: 1024 auto_replay_nacks: true grpc: reconnect_base_delay: 500ms reconnect_max_delay: 30s reconnect_max_attempts: 0 shutdown_timeout: 10s buffer_size: 1000 http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#metadata)Metadata Every emitted message has: - `topic`: The full Pub/Sub topic path (for example, "/event/Order\_\_e"). - `replay_id`: The Pub/Sub replay ID in hex (streaming events only). CDC events also carry: - `operation`: "read" for snapshot rows; "create", "update", "delete", or "undelete" for CDC events. - `sobject`: The sObject API name (for example, "Account"). - `record_ids`: Comma-separated record IDs affected by the event (when present). Platform Events also carry: - `event_uuid`: The Salesforce `EventUuid` extracted from the payload (the canonical dedup key), when present. ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org; check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` Name of the cache resource used to persist snapshot cursor and per-topic replay IDs across restarts. The cache must be declared under the top-level `cache_resources` block. Choose a durable cache (Redis, Postgres, DynamoDB) for production; in-memory caches lose checkpoints on restart. **Type**: `string` ```yaml # Examples: checkpoint_cache: persistent_cache ``` ### [](#checkpoint_cache_key)`checkpoint_cache_key` Key inside the checkpoint cache where this input’s state is stored. Change when running multiple `salesforce_cdc` inputs against the same cache resource to avoid collisions. **Type**: `string` **Default**: `salesforce_cdc` ### [](#checkpoint_limit)`checkpoint_limit` Maximum number of unacknowledged batches in flight (per topic) before that topic pauses reading. Prevents unbounded memory growth when downstream components stall. Higher values increase throughput in steady state; lower values bound memory under backpressure. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive; prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#grpc)`grpc` gRPC transport tuning for the Pub/Sub API connection. **Type**: `object` ### [](#grpc-buffer_size)`grpc.buffer_size` Size of the internal gRPC event receive buffer. **Type**: `int` **Default**: `1000` ### [](#grpc-reconnect_base_delay)`grpc.reconnect_base_delay` Base delay for gRPC reconnection backoff. **Type**: `string` **Default**: `500ms` ### [](#grpc-reconnect_max_attempts)`grpc.reconnect_max_attempts` Maximum number of gRPC reconnection attempts. 0 means unlimited. This budget also governs transient schema-fetch failures: an event whose schema fetch keeps failing at the same replay position is retried (by reconnect-and-redeliver, or inline before the first event is delivered) this many times before the topic fails permanently. Deterministic schema failures - a schema that is missing, inaccessible, or fails to compile - and payloads that repeatedly fail to decode give up after a small fixed number of attempts regardless of this setting, since retrying cannot change the outcome. **Type**: `int` **Default**: `0` ### [](#grpc-reconnect_max_delay)`grpc.reconnect_max_delay` Maximum delay for gRPC reconnection backoff. **Type**: `string` **Default**: `30s` ### [](#grpc-shutdown_timeout)`grpc.shutdown_timeout` Timeout for graceful gRPC client shutdown. **Type**: `string` **Default**: `10s` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#max_parallel_snapshot_objects)`max_parallel_snapshot_objects` Number of sObjects snapshotted concurrently during the REST snapshot phase. Each in-flight snapshot consumes one HTTP connection and Salesforce API call quota. Default 1 serializes the work; raise when snapshotting many sObjects and your API limits permit. **Type**: `int` **Default**: `1` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#replay_preset)`replay_preset` Initial replay position used per topic only on first run (when no checkpoint exists in the cache); ignored once a topic’s replay ID has been written. - `latest`: Start from new events only; any changes between prior run and Connect are skipped. - `earliest`: Replay from the retention start (24h standard, 72h with enhanced retention). Use to recover missed events after outages. **Type**: `string` **Default**: `latest` **Options**: `latest`, `earliest` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` Page size for the REST snapshot query; records per `/query` response. Must be between 200 and 2000 per Salesforce REST API limits. Larger pages reduce HTTP round trips; smaller pages reduce peak memory per fetch. **Type**: `int` **Default**: `2000` ```yaml # Examples: snapshot_max_batch_size: 2000 # --- snapshot_max_batch_size: 500 ``` ### [](#stream_batch_size)`stream_batch_size` Number of events requested per gRPC `Fetch` call, per topic. Higher values improve throughput at the cost of peak batch memory; lower values give steadier latency under load. **Type**: `int` **Default**: `100` ```yaml # Examples: stream_batch_size: 100 # --- stream_batch_size: 500 ``` ### [](#stream_snapshot)`stream_snapshot` When true (default), paginate a full REST snapshot of every CDC sObject in `topics` before opening any streaming subscription. When false, skip the snapshot and start streaming immediately. Platform Event topics (`/event/…​`) are always skipped; they have no REST equivalent. **Type**: `bool` **Default**: `true` ### [](#topics)`topics[]` Pub/Sub topics to subscribe to. Each entry is one of: a bare sObject name (`Account` → `/data/AccountChangeEvent`), an explicit CDC channel (`/data/AccountChangeEvent`), the CDC firehose (`/data/ChangeEvents`), or a Platform Event topic (`/event/Order__e`, `/event/LoginEventStream`). Each topic gets its own gRPC subscription with an independent replay cursor. **Type**: `array` ```yaml # Examples: topics: - Account - Contact # --- topics: - /data/ChangeEvents # --- topics: - Account - /event/Order__e # --- topics: - Opportunity - MyCustom__c - /event/Sync_Requested__e ``` ## [](#examples)Examples ### [](#snapshot-cdc-for-core-objects)Snapshot + CDC for core objects Snapshot Accounts, Contacts, and Opportunities, then stream their CDC events: ```yaml input: salesforce_cdc: org_url: https://acme.my.salesforce.com client_id: ${SALESFORCE_CLIENT_ID} client_secret: ${SALESFORCE_CLIENT_SECRET} topics: [ Account, Contact, Opportunity ] stream_snapshot: true replay_preset: latest checkpoint_cache: persistent_cache cache_resources: - label: persistent_cache redis: url: redis://localhost:6379 ``` ### [](#cdc-firehose-without-snapshot)CDC firehose without snapshot Subscribe to every CDC-enabled sObject via the firehose, no historical snapshot: ```yaml input: salesforce_cdc: org_url: https://acme.my.salesforce.com client_id: ${SALESFORCE_CLIENT_ID} client_secret: ${SALESFORCE_CLIENT_SECRET} topics: [ /data/ChangeEvents ] stream_snapshot: false replay_preset: latest checkpoint_cache: persistent_cache ``` ### [](#mixed-cdc-platform-events)Mixed CDC + Platform Events Combine Account CDC with a custom Platform Event in one pipeline: ```yaml input: salesforce_cdc: org_url: https://acme.my.salesforce.com client_id: ${SALESFORCE_CLIENT_ID} client_secret: ${SALESFORCE_CLIENT_SECRET} topics: - Account - /event/Order_Created__e stream_snapshot: true replay_preset: latest checkpoint_cache: persistent_cache ``` ### [](#platform-events-only)Platform Events only Stream a custom event and a standard login stream: snapshot is skipped automatically: ```yaml input: salesforce_cdc: org_url: https://acme.my.salesforce.com client_id: ${SALESFORCE_CLIENT_ID} client_secret: ${SALESFORCE_CLIENT_SECRET} topics: - /event/Order__e - /event/LoginEventStream replay_preset: latest checkpoint_cache: persistent_cache ``` --- # Page 88: salesforce_graphql **URL**: https://docs.redpanda.com/connect/components/inputs/salesforce_graphql.md --- # salesforce_graphql --- title: salesforce_graphql latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/salesforce_graphql page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/salesforce_graphql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/salesforce_graphql.adoc description: Executes a GraphQL query against the Salesforce UIAPI and emits one message per record. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Executes a GraphQL query against the Salesforce UIAPI (`POST /services/data/{api_version}/graphql`), walks the response tree, and emits one message per record. When records are exhausted the input shuts down, letting the pipeline terminate gracefully (or the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to take over). Introduced in version 4.90.3. ## [](#when-to-use-this-input)When to use this input Use `salesforce_graphql` for: - Cross-object queries in a single request (parent + children + grandchildren). - Response shapes that already match your downstream schema. - Queries benefiting from GraphQL’s field-level selection and nesting. - Adopting Salesforce’s UIAPI / future-forward query surface. Use a different Salesforce input instead if: - You only need single-object SELECTs: use [`salesforce`](https://docs.redpanda.com/connect/components/inputs/salesforce/) (simpler, no GraphQL schema knowledge needed). - You need continuous change events: use [`salesforce_cdc`](https://docs.redpanda.com/connect/components/inputs/salesforce_cdc/). When the query selects an `edges`/`pageInfo` connection, the input transparently paginates by injecting `after: ""` into the query string between requests. The query must include `pageInfo { hasNextPage endCursor }` for pagination to terminate cleanly. Responses without an `edges` array are emitted as a single message and the input completes. ### Common ```yml input: label: "" salesforce_graphql: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 query: "" # No default (required) variables: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" salesforce_graphql: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 query: "" # No default (required) variables: "" # No default (optional) auto_replay_nacks: true http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#metadata)Metadata This input adds no Salesforce-specific metadata. GraphQL response shapes vary by query, so record identity and context travel in the message body. ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org; check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive; prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#query)`query` The GraphQL query document as a single string. Must target the Salesforce UIAPI schema (`uiapi.query.` **or `uiapi.mutation.`**). Typically follows the UIAPI convention of nested `edges { node { field { value } } }`. Variables are referenced with `$name` and supplied via the `variables` field. For automatic pagination, include `pageInfo { hasNextPage endCursor }` in the relevant connection. **Type**: `string` ```yaml # Examples: query: query Accounts { uiapi { query { Account { edges { node { Id { value } Name { value } } } pageInfo { hasNextPage endCursor } } } } } # --- query: "query Accounts($first: Int) { uiapi { query { Account(first: $first) { edges { node { Id { value } Name { value } } } pageInfo { hasNextPage endCursor } } } } }" ``` ### [](#variables)`variables` Optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) whose result must be an object whose keys are GraphQL variable names referenced by `query`. Values pass through as JSON in the request body’s `variables` field. The mapping runs once at startup with no message context, so it can use literal values and functions that don’t read a message, such as `now()`, but can’t reference message content or metadata. **Type**: `string` ```yaml # Examples: variables: 'root = {"first": 100}' # --- variables: 'root = {"since": now().ts_format("2006-01-02T15:04:05Z"), "limit": 500}' ``` ## [](#examples)Examples ### [](#account-snapshot-via-graphql)Account snapshot via GraphQL Fetch the first 500 Accounts with a nested UIAPI query: ```yaml input: salesforce_graphql: org_url: https://acme.my.salesforce.com client_id: ${SALESFORCE_CLIENT_ID} client_secret: ${SALESFORCE_CLIENT_SECRET} query: | query Accounts($limit: Int) { uiapi { query { Account(first: $limit) { edges { node { Id { value } Name { value } LastModifiedDate { value } } } pageInfo { hasNextPage endCursor } } } } } variables: | root = {"limit": 500} ``` --- # Page 89: salesforce **URL**: https://docs.redpanda.com/connect/components/inputs/salesforce.md --- # salesforce --- title: salesforce latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/salesforce page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/salesforce.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/salesforce.adoc description: Runs a SOQL query against the Salesforce REST API, paginates through all result pages, and emits one message per record. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Executes a Salesforce SOQL query once and emits a message for each record. Introduced in version 4.90.3. Runs a SOQL query against the Salesforce REST API, paginates through all result pages, and emits one message per record. When results are exhausted the input shuts down, letting the pipeline terminate gracefully (or the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to take over). ## [](#when-to-use-this-input)When to use this input Use `salesforce` for: - One-shot extracts (for example, dump all Accounts into a warehouse). - Periodic full-table refreshes via a scheduled pipeline or [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/). - Backfills and ad-hoc queries. - Warming up a downstream pipeline before switching to CDC. Use a different Salesforce input instead if: - You need continuous change events: use [`salesforce_cdc`](https://docs.redpanda.com/connect/components/inputs/salesforce_cdc/). - You need a GraphQL query (cross-object in one request): use [`salesforce_graphql`](https://docs.redpanda.com/connect/components/inputs/salesforce_graphql/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `sobject`: The sObject API name (for example, "Account"). ## [](#authentication)Authentication Uses the Salesforce OAuth 2.0 Client Credentials flow. Create a Connected App in Salesforce, enable OAuth settings and the Client Credentials Flow, then supply the Consumer Key as `client_id` and Consumer Secret as `client_secret`. ### Common ```yml input: label: "" salesforce: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 object: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" salesforce: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 object: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) auto_replay_nacks: true http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org; check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#args_mapping)`args_mapping` Optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) whose result must be an array of values matching the count of `?` placeholders in `where`. Values are SOQL-escaped: strings become quoted literals, timestamps become ISO-8601, booleans and numbers pass through. The mapping runs once at startup with no message context, so it can use literal values and functions that don’t read a message, such as `now()`, but can’t reference message content or metadata. **Type**: `string` ```yaml # Examples: args_mapping: root = [ (now() - "1h").ts_format("2006-01-02T15:04:05Z") ] # --- args_mapping: root = [ "Active", (now() - "24h").ts_format("2006-01-02T15:04:05Z") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive; prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#columns)`columns[]` Ordered list of field API names to retrieve. SOQL does not accept `*`; every field must be listed explicitly. Standard fields use their documented names; custom fields end with `__c`. Relationship fields traverse parents via dot notation (`Account.Name`, `Owner.Manager.Email`) up to 5 levels deep. Requesting a non-existent or non-queryable field fails at Connect time with a SOQL compile error. **Type**: `array` ```yaml # Examples: columns: - Id - Name - LastModifiedDate # --- columns: - Id - Account.Name - Owner.Email # --- columns: - Id - MyCustom__c ``` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#object)`object` The sObject API name to SELECT from. Case-sensitive; uses the API name, not the display label. Standard objects use the noun (`Account`, `Opportunity`); custom objects end with `__c`; Big Objects end with `__b`; External Objects end with `__x`. Confirm the exact API name in Setup → Object Manager. **Type**: `string` ```yaml # Examples: object: Account # --- object: Contact # --- object: MyCustom__c ``` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#prefix)`prefix` Optional SOQL fragment inserted before the SELECT keyword. Rarely needed; provided for forward compatibility with future SOQL extensions or Bulk API framing. **Type**: `string` ### [](#suffix)`suffix` Optional SOQL fragment appended after the WHERE clause. Typical uses: `ORDER BY` for deterministic pagination, `LIMIT` to cap result size, `FOR REFERENCE` / `FOR VIEW` to mark records for Chatter tracking. **Type**: `string` ```yaml # Examples: suffix: ORDER BY LastModifiedDate DESC # --- suffix: ORDER BY Id LIMIT 1000 # --- suffix: ORDER BY CreatedDate DESC LIMIT 10000 ``` ### [](#where)`where` Optional SOQL WHERE body, without the `WHERE` keyword. `?` placeholders are substituted client-side from `args_mapping` with SOQL literal escaping (quoted strings, ISO-8601 datetimes). Supports the full WHERE grammar: `AND`/`OR`/`NOT`, `LIKE`, `IN`, date literals (`TODAY`, `LAST_N_DAYS:7`), subqueries. Date/datetime comparisons require ISO-8601 with explicit timezone. **Type**: `string` ```yaml # Examples: where: LastModifiedDate > ? # --- where: Status__c = ? AND CreatedDate > ? # --- where: OwnerId IN (?, ?) ``` ## [](#examples)Examples ### [](#recent-account-changes)Recent Account Changes Query Accounts modified in the last hour and emit one message per record: ```yaml input: salesforce: org_url: https://acme.my.salesforce.com client_id: ${SALESFORCE_CLIENT_ID} client_secret: ${SALESFORCE_CLIENT_SECRET} object: Account columns: [ Id, Name, LastModifiedDate ] where: LastModifiedDate > ? args_mapping: | root = [ (now() - "1h").ts_format("2006-01-02T15:04:05Z") ] ``` --- # Page 90: sap_hana **URL**: https://docs.redpanda.com/connect/components/inputs/sap_hana.md --- # sap_hana --- title: sap_hana latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/sap_hana page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sap_hana.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sap_hana.adoc description: Reads rows from a SAP HANA table. page-git-created-date: "2026-10-15" page-git-modified-date: "2026-10-15" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Reads rows from a SAP HANA table. Reads rows from a SAP HANA table. Supports five modes: - `bulk`: reads all rows once then the input terminates (use with [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) for periodic re-reads). - `incrementing`: polls for rows where `incrementing_column` exceeds the last seen value, emitting only net-new rows. - `query`: executes a user-supplied SQL statement and emits one message per result row. - `timestamp`: polls for rows where `timestamp_column` falls within `(last_hwm, database_now - timestamp_delay]`. The bound is read from the database clock (see `timestamp_clock`) and the delay absorbs commit lag. The HWM advances to the window bound once the window is fully consumed, so a restart mid-window re-reads that window from its start (rows sharing a timestamp cannot be split, hence no finer checkpoint). - `timestamp+incrementing`: like `timestamp` but orders by `(timestamp_column, incrementing_column)` and resumes after the exact `(timestamp, incrementing)` pair of the last delivered row, so rows sharing a timestamp are neither duplicated nor missed and a restart mid-window continues from the last acknowledged batch rather than the window start. ## [](#metadata)Metadata Messages produced in `bulk`, `incrementing`, `timestamp`, and `timestamp+incrementing` modes carry the following metadata fields (`query` mode attaches none): - `table_name`: The HANA table name. - `database_schema`: The configured `schema_name`. Only present when `schema_name` is set. - `schema`: Avro-compatible schema derived from `SYS.TABLE_COLUMNS`, suitable for use with `schema_registry_encode`. Column additions are detected automatically without a pipeline restart. Only present when `schema_name` is configured. - `primary_key_columns`: JSON array of the table’s primary-key column names in key order. Only present when `schema_name` is configured and the table has a primary key. ## [](#troubleshooting)Troubleshooting **`schema` and `primary_key_columns` metadata are missing.** The input reads them from `SYS.TABLE_COLUMNS` and `SYS.INDEX_COLUMNS`; when that read fails it logs a warning and carries on without them, and `incrementing_initial_value` falls back to a guessed bind type. Check the warning in the logs and grant the connecting user `CATALOG READ` (or `SELECT` on the table’s schema), and make sure `schema_name` is set. **"invalid table name" or "invalid column name" for a table that exists.** Identifiers in this config are quoted when sent to HANA, so they must match the catalog’s case exactly. HANA upper-cases identifiers that were created without quotes: a table created as `create table orders` is `ORDERS` in the catalog and must be configured as such. **Batches arrive one at a time.** `fetch_size` larger than `checkpoint_limit` means a batch cannot be handed on until the previous one is acknowledged (a warning is logged at startup). Raise `checkpoint_limit` or lower `fetch_size`. **Rows written around the time of a poll are missing in timestamp modes.** The poll window ends at the database clock minus `timestamp_delay`; a row whose timestamp was assigned before that bound but committed after the poll is never seen. Set `timestamp_delay` longer than your longest write transaction, and pick the `timestamp_clock` that matches how the column is populated. ### Common ```yml input: label: "" sap_hana: dsn: "" # No default (required) schema_name: "" # No default (optional) table: "" # No default (optional) mode: bulk query: "" # No default (optional) incrementing_column: "" # No default (optional) incrementing_initial_value: "" poll_interval: 60s timestamp_column: "" # No default (optional) timestamp_initial_value: "" timestamp_delay: 5s numeric_mapping: none checkpoint_cache: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml input: label: "" sap_hana: dsn: "" # No default (required) fetch_size: 128 schema_name: "" # No default (optional) table: "" # No default (optional) mode: bulk query: "" # No default (optional) incrementing_column: "" # No default (optional) incrementing_initial_value: "" poll_interval: 60s timestamp_column: "" # No default (optional) timestamp_initial_value: "" timestamp_delay: 5s timestamp_clock: database numeric_mapping: none max_retries: 3 retry_backoff: 1s checkpoint_cache: "" # No default (optional) checkpoint_cache_key: sap_hana_hwm checkpoint_limit: 1024 auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#checkpoint_cache)`checkpoint_cache` Name of a cache resource to persist the high-water mark across restarts. When set, the connector resumes from where it left off rather than starting from scratch. Only effective in `incrementing`, `timestamp`, and `timestamp+incrementing` modes. The cache must be declared under `cache_resources`. Choose a durable backend (Redis, PostgreSQL) for production; in-memory caches lose state on restart. **Type**: `string` ```yaml # Examples: checkpoint_cache: redis_cache ``` ### [](#checkpoint_cache_key)`checkpoint_cache_key` Key used to store the checkpoint in `checkpoint_cache`. Change this when multiple `sap_hana` inputs share the same cache resource to avoid key collisions. **Type**: `string` **Default**: `sap_hana_hwm` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be in flight (read but not yet acknowledged) at a given time. The high-water mark is only checkpointed once every message before it has been acknowledged, preserving at-least-once delivery even when batches are acknowledged out of order. When `fetch_size` exceeds this limit, batches are delivered one at a time (each waits for the previous batch’s acknowledgement), so raise this alongside large `fetch_size` values to keep batches flowing concurrently. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` SAP HANA connection DSN in `hdb://user:password@host:port` form. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: dsn: hdb://user:password@host:39017 ``` ### [](#fetch_size)`fetch_size` Number of rows requested per FetchNext round-trip. Larger values reduce round-trips on high-latency connections. **Type**: `int` **Default**: `128` ### [](#incrementing_column)`incrementing_column` Column to use as the high-water mark for `incrementing` mode. Must be strictly monotonically increasing with no duplicate values — a BIGINT auto-increment column is ideal. Columns that produce duplicate values (e.g. a plain TIMESTAMP) will cause rows whose value ties span a `fetch_size` boundary to be re-delivered. Use `timestamp+incrementing` mode to handle timestamp columns safely. **Type**: `string` ### [](#incrementing_initial_value)`incrementing_initial_value` Initial high-water mark value. When empty, all existing rows are emitted on the first run. The value is converted to the ``incrementing_column’s type from the catalog on connect: integers for integer columns, RFC3339 or `YYYY-MM-DD[ HH:MM:SS]`` for DATE/TIMESTAMP columns, hexadecimal for BINARY/VARBINARY columns, and the literal string (leading zeros preserved) for character columns. A persisted checkpoint takes precedence over this value. **Type**: `string` **Default**: `""` ### [](#max_retries)`max_retries` Maximum number of times to retry a failed query before returning an error. Set to `0` to disable retries. **Type**: `int` **Default**: `3` ### [](#mode)`mode` Operation mode. **Type**: `string` **Default**: `bulk` **Options**: `bulk`, `incrementing`, `query`, `timestamp`, `timestamp+incrementing` ### [](#numeric_mapping)`numeric_mapping` Controls how DECIMAL/NUMERIC columns are emitted: - `none`: emit as a canonical decimal string preserving full precision, except integer-typed columns (`DECIMAL(p,0)` with precision ⇐ 18) which are emitted as integers. - `best_fit`: additionally emit columns whose precision fits a double (precision ⇐ 15) as floating-point numbers; wider values fall back to canonical decimal strings. **Type**: `string` **Default**: `none` **Options**: `none`, `best_fit` ### [](#poll_interval)`poll_interval` How long to wait between polls in `incrementing`, `timestamp`, and `timestamp+incrementing` modes. **Type**: `string` **Default**: `60s` ```yaml # Examples: poll_interval: 10s # --- poll_interval: 5m ``` ### [](#query)`query` Custom SQL statement to execute. Only used when `mode` is `query`. **Type**: `string` ### [](#retry_backoff)`retry_backoff` Base delay between query retries. The delay grows linearly with the attempt number (attempt N waits N times this value). **Type**: `string` **Default**: `1s` ### [](#schema_name)`schema_name` Database schema for the table. When set, an Avro-compatible `schema` metadata field is attached to every message using data from `SYS.TABLE_COLUMNS`. **Type**: `string` ### [](#table)`table` Table to read from. Required when `mode` is `bulk` or `incrementing`. **Type**: `string` ### [](#timestamp_clock)`timestamp_clock` Which database clock bounds each timestamp-mode poll window. The bound is always read from HANA, never from the connector host, so it shares the clock and timezone convention of the column it is compared against: - `database`: `CURRENT_TIMESTAMP` (session timezone), matching columns populated by `DEFAULT CURRENT_TIMESTAMP` or `NOW()`. - `database_utc`: `CURRENT_UTCTIMESTAMP`, for columns populated with UTC values. **Type**: `string` **Default**: `database` **Options**: `database`, `database_utc` ### [](#timestamp_column)`timestamp_column` Column to use as the high-water mark for `timestamp` and `timestamp+incrementing` modes. Must be a TIMESTAMP or LONGDATE column. **Type**: `string` ### [](#timestamp_delay)`timestamp_delay` Commit-lag buffer for timestamp modes. The upper bound for each poll is the database clock minus `timestamp_delay`, so rows whose timestamp was assigned slightly before a still-uncommitted transaction finished are not missed. **Type**: `string` **Default**: `5s` ```yaml # Examples: timestamp_delay: 0s # --- timestamp_delay: 30s ``` ### [](#timestamp_initial_value)`timestamp_initial_value` Initial high-water mark in RFC3339 format (e.g. `2024-01-01T00:00:00Z`). When empty, all existing rows are emitted on the first run. **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#incremental-reads-with-a-durable-checkpoint)Incremental reads with a durable checkpoint Polls an orders table for new rows by primary key every 30 seconds and persists the high-water mark in a file cache so a restart resumes where it left off. ```yaml input: sap_hana: dsn: hdb://user:password@hana-host:39017 mode: incrementing schema_name: SALES table: ORDERS incrementing_column: ORDER_ID poll_interval: 30s checkpoint_cache: hana_checkpoints cache_resources: - label: hana_checkpoints file: directory: /var/lib/redpanda-connect/sap_hana ``` ### [](#change-tracking-by-timestamp-with-automatic-avro-schema-registration)Change tracking by timestamp with automatic Avro schema registration Follows an updated-at column together with the primary key so rows sharing a timestamp are neither duplicated nor missed, bounds each poll by the database’s UTC clock with a commit-lag buffer, and registers the table’s schema from the emitted metadata before encoding. ```yaml input: sap_hana: dsn: hdb://user:password@hana-host:39017 mode: timestamp+incrementing schema_name: SALES table: ORDERS timestamp_column: UPDATED_AT incrementing_column: ORDER_ID timestamp_clock: database_utc timestamp_delay: 30s poll_interval: 10s checkpoint_cache: hana_checkpoints pipeline: processors: - schema_registry_encode: url: http://schema-registry:8081 subject: sales.orders-value schema_metadata: schema format: avro avro: raw_json: true output: redpanda: seed_brokers: [ "broker:9092" ] topic: sales.orders cache_resources: - label: hana_checkpoints file: directory: /var/lib/redpanda-connect/sap_hana ``` --- # Page 91: schema_registry **URL**: https://docs.redpanda.com/connect/components/inputs/schema_registry.md --- # schema_registry --- title: schema_registry latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/schema_registry page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/schema_registry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/schema_registry.adoc description: Reads schemas from SchemaRegistry. page-git-created-date: "2024-09-04" page-git-modified-date: "2026-10-10" --- Reads schemas from a schema registry. You can use this connector to extract and back up schemas during a data migration. This input uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). Introduced in version 4.32.2. #### Common ```yml input: label: "" schema_registry: url: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" schema_registry: url: "" # No default (required) include_deleted: false subject_filter: "" fetch_in_order: true tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `schema_registry_subject` - `schema_registry_subject_compatibility_level` - `schema_registry_version` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#read-schemas)Read schemas Read all schemas (including deleted) from a Schema Registry instance which are associated with subjects matching the `^foo.*` filter. ```yaml input: schema_registry: url: http://localhost:8081 include_deleted: true subject_filter: ^foo.* ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#fetch_in_order)`fetch_in_order` Indicate whether to fetch all schemas from the schema registry service and sort them by ID. Set this value to `true` if you use schemas that refer to other schemas (schema references). Requires version 4.37.0 or later. **Type**: `bool` **Default**: `true` ### [](#include_deleted)`include_deleted` Include deleted entities. **Type**: `bool` **Default**: `false` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#subject_filter)`subject_filter` Include only subjects which match the regular expression filter, or leave this field value blank to select all subjects. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 92: sequence **URL**: https://docs.redpanda.com/connect/components/inputs/sequence.md --- # sequence --- title: sequence latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/sequence page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sequence.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sequence.adoc description: Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next, and so on. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next, and so on. This input is useful for consuming from inputs that have an explicit end but must not be consumed in parallel. #### Common ```yml input: label: "" sequence: inputs: [] # No default (required) ``` #### Advanced ```yml input: label: "" sequence: sharded_join: type: none id_path: "" iterations: 1 merge_strategy: array inputs: [] # No default (required) ``` ## [](#examples)Examples ### [](#end-of-stream-message)End of Stream Message A common use case for sequence might be to generate a message at the end of our main input. With the following config once the records within `./dataset.csv` are exhausted our final payload `{"status":"finished"}` will be routed through the pipeline. ```yaml input: sequence: inputs: - file: paths: [ ./dataset.csv ] scanner: csv: {} - generate: count: 1 mapping: 'root = {"status":"finished"}' ``` ### [](#joining-data-simple)Joining Data (Simple) Redpanda Connect can be used to join unordered data from fragmented datasets in memory by specifying a common identifier field and a number of sharded iterations. For example, given two CSV files, the first called "main.csv", which contains rows of user data: ```csv uuid,name,age AAA,Melanie,34 BBB,Emma,28 CCC,Geri,45 ``` And the second called "hobbies.csv" that, for each user, contains zero or more rows of hobbies: ```csv uuid,hobby CCC,pokemon go AAA,rowing AAA,golf ``` We can parse and join this data into a single dataset: ```json {"uuid":"AAA","name":"Melanie","age":34,"hobbies":["rowing","golf"]} {"uuid":"BBB","name":"Emma","age":28} {"uuid":"CCC","name":"Geri","age":45,"hobbies":["pokemon go"]} ``` With the following config: ```yaml input: sequence: sharded_join: type: full-outer id_path: uuid merge_strategy: array inputs: - file: paths: - ./hobbies.csv - ./main.csv scanner: csv: {} ``` ### [](#joining-data-advanced)Joining Data (Advanced) In this example we are able to join unordered and fragmented data from a combination of CSV files and newline-delimited JSON documents by specifying multiple sequence inputs with their own processors for extracting the structured data. The first file "main.csv" contains straight forward CSV data: ```csv uuid,name,age AAA,Melanie,34 BBB,Emma,28 CCC,Geri,45 ``` And the second file called "hobbies.ndjson" contains JSON documents, one per line, that associate an identifier with an array of hobbies. However, these data objects are in a nested format: ```json {"document":{"uuid":"CCC","hobbies":[{"type":"pokemon go"}]}} {"document":{"uuid":"AAA","hobbies":[{"type":"rowing"},{"type":"golf"}]}} ``` And so we will want to map these into a flattened structure before the join, and then we will end up with a single dataset that looks like this: ```json {"uuid":"AAA","name":"Melanie","age":34,"hobbies":["rowing","golf"]} {"uuid":"BBB","name":"Emma","age":28} {"uuid":"CCC","name":"Geri","age":45,"hobbies":["pokemon go"]} ``` With the following config: ```yaml input: sequence: sharded_join: type: full-outer id_path: uuid iterations: 10 merge_strategy: array inputs: - file: paths: [ ./main.csv ] scanner: csv: {} - file: paths: [ ./hobbies.ndjson ] scanner: lines: {} processors: - mapping: | root.uuid = this.document.uuid root.hobbies = this.document.hobbies.map_each(this.type) ``` ## [](#fields)Fields ### [](#inputs)`inputs[]` An array of inputs to read from sequentially. **Type**: `array` ### [](#sharded_join)`sharded_join` EXPERIMENTAL: Provides a way to perform outer joins of arbitrarily structured and unordered data resulting from the input sequence, even when the overall size of the data surpasses the memory available on the machine. When configured the sequence of inputs will be consumed one or more times according to the number of iterations, and when more than one iteration is specified each iteration will process an entirely different set of messages by sharding them by the ID field. Increasing the number of iterations reduces the memory consumption at the cost of needing to fully parse the data each time. Each message must be structured (JSON or otherwise processed into a structured form) and the fields will be aggregated with those of other messages sharing the ID. At the end of each iteration the joined messages are flushed downstream before the next iteration begins, hence keeping memory usage limited. Requires version 3.40.0 or later. **Type**: `object` ### [](#sharded_join-id_path)`sharded_join.id_path` A [dot path](https://docs.redpanda.com/connect/configuration/field_paths/) that points to a common field within messages of each fragmented data set and can be used to join them. Messages that are not structured or are missing this field will be dropped. This field must be set in order to enable joins. **Type**: `string` **Default**: `""` ### [](#sharded_join-iterations)`sharded_join.iterations` The total number of iterations (shards), increasing this number will increase the overall time taken to process the data, but reduces the memory used in the process. The real memory usage required is significantly higher than the real size of the data and therefore the number of iterations should be at least an order of magnitude higher than the available memory divided by the overall size of the dataset. **Type**: `int` **Default**: `1` ### [](#sharded_join-merge_strategy)`sharded_join.merge_strategy` The chosen strategy to use when a data join would otherwise result in a collision of field values. The strategy `array` means non-array colliding values are placed into an array and colliding arrays are merged. The strategy `replace` replaces old values with new values. The strategy `keep` keeps the old value. **Type**: `string` **Default**: `array` **Options**: `array`, `replace`, `keep` ### [](#sharded_join-type)`sharded_join.type` The type of join to perform. A `full-outer` ensures that all identifiers seen in any of the input sequences are sent, and is performed by consuming all input sequences before flushing the joined results. An `outer` join consumes all input sequences but only writes data joined from the last input in the sequence, similar to a left or right outer join. With an `outer` join if an identifier appears multiple times within the final sequence input it will be flushed each time it appears. `full-outter` and `outter` have been deprecated in favour of `full-outer` and `outer`. **Type**: `string` **Default**: `none` **Options**: `none`, `full-outer`, `outer`, `full-outter`, `outter` --- # Page 93: sftp **URL**: https://docs.redpanda.com/connect/components/inputs/sftp.md --- # sftp --- title: sftp latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/sftp page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sftp.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sftp.adoc description: Consumes files from an SFTP server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes files from an SFTP server. Introduced in version 3.39.0. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `sftp_path` - `sftp_mod_time` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ### Common ```yml input: label: "" sftp: address: "" # No default (required) credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" paths: [] # No default (required) auto_replay_nacks: true scanner: to_the_end: {} watcher: enabled: false minimum_age: 1s poll_interval: 1s cache: "" ``` ### Advanced ```yml input: label: "" sftp: address: "" # No default (required) connection_timeout: 30s credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" ssh_algorithms: additional_key_exchanges: [] # No default (optional) additional_ciphers: [] # No default (optional) additional_macs: [] # No default (optional) max_sftp_sessions: 10 paths: [] # No default (required) auto_replay_nacks: true scanner: to_the_end: {} delete_on_finish: false watcher: enabled: false minimum_age: 1s poll_interval: 1s cache: "" ``` ## [](#fields)Fields ### [](#address)`address` The address (hostname or IP address) of the SFTP server to connect to. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#connection_timeout)`connection_timeout` The connection timeout to use when connecting to the target server. Requires version 4.59.0 or later. **Type**: `string` **Default**: `30s` ### [](#credentials)`credentials` The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access. **Type**: `object` ### [](#credentials-host_public_key)`credentials.host_public_key` The raw contents of the SFTP server’s public key, used for host key verification. Requires version 4.59.0 or later. **Type**: `string` ### [](#credentials-host_public_key_file)`credentials.host_public_key_file` The path to the SFTP server’s public key file, used for host key verification. Requires version 4.59.0 or later. **Type**: `string` ### [](#credentials-password)`credentials.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-private_key)`credentials.private_key` The raw contents of the private key used to authenticate with the SFTP server. This field provides an alternative to `private_key_file`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.51.0 or later. **Type**: `string` ### [](#credentials-private_key_file)`credentials.private_key_file` The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the `private_key` field. **Type**: `string` ### [](#credentials-private_key_pass)`credentials.private_key_pass` An optional passphrase for decrypting the private key, if it is encrypted. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-ssh_algorithms)`credentials.ssh_algorithms` Additional SSH transport algorithms to permit for compatibility with legacy SFTP servers. Configured algorithms are appended to the default algorithms, which remain preferred, and when unset the defaults are used unchanged. Algorithms the underlying SSH library classifies as insecure weaken transport security and should only be enabled when required by a known server. These settings do not affect host key verification. Requires version 4.113.0 or later. **Type**: `object` ### [](#credentials-ssh_algorithms-additional_ciphers)`credentials.ssh_algorithms.additional_ciphers[]` Ciphers to permit in addition to the defaults, for example `aes128-cbc`. **Type**: `array` ### [](#credentials-ssh_algorithms-additional_key_exchanges)`credentials.ssh_algorithms.additional_key_exchanges[]` Key exchange algorithms to permit in addition to the defaults, for example `diffie-hellman-group1-sha1` or `diffie-hellman-group-exchange-sha1`. **Type**: `array` ### [](#credentials-ssh_algorithms-additional_macs)`credentials.ssh_algorithms.additional_macs[]` MAC algorithms to permit in addition to the defaults. **Type**: `array` ### [](#credentials-username)`credentials.username` The username required to authenticate with the SFTP server. **Type**: `string` **Default**: `""` ### [](#delete_on_finish)`delete_on_finish` Whether to delete files from the server once they are processed. **Type**: `bool` **Default**: `false` ### [](#max_sftp_sessions)`max_sftp_sessions` The maximum number of SFTP sessions. Requires version 4.59.0 or later. **Type**: `int` **Default**: `10` ### [](#paths)`paths[]` A list of paths to consume sequentially. Glob patterns are supported. **Type**: `array` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#watcher)`watcher` An experimental mode whereby the input will periodically scan the target paths for new files and consume them, when all files are consumed the input will continue polling for new files. Requires version 3.42.0 or later. **Type**: `object` ### [](#watcher-cache)`watcher.cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) for storing the paths of files already consumed. **Type**: `string` **Default**: `""` ### [](#watcher-enabled)`watcher.enabled` Whether file watching is enabled. **Type**: `bool` **Default**: `false` ### [](#watcher-minimum_age)`watcher.minimum_age` The minimum period of time since a file was last updated before attempting to consume it. Increasing this period decreases the likelihood that a file will be consumed whilst it is still being written to. **Type**: `string` **Default**: `1s` ```yaml # Examples: minimum_age: 10s # --- minimum_age: 1m # --- minimum_age: 10m ``` ### [](#watcher-poll_interval)`watcher.poll_interval` The interval between each attempt to scan the target paths for new files. **Type**: `string` **Default**: `1s` ```yaml # Examples: poll_interval: 100ms # --- poll_interval: 1s ``` --- # Page 94: slack_users **URL**: https://docs.redpanda.com/connect/components/inputs/slack_users.md --- # slack_users --- title: slack_users latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/slack_users page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/slack_users.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/slack_users.adoc description: Returns the full profile of all users in your Slack organization using the users.list API method. You can filter the returned users by team ID. page-git-created-date: "2025-05-02" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Returns [the full profile](https://api.slack.com/methods/users.list#examples) of all users in your Slack organization using the API method [users.list](https://api.slack.com/methods/users.list). Optionally, you can filter the list of returned users by team ID. This input is useful when you need to: - Join user information to Slack posts. - Ingest user information into a data lakehouse to create joins with other fields. Introduced in version 4.52.0. ```yml input: label: "" slack_users: bot_token: "" # No default (required) team_id: "" auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` Your [Slack bot user’s OAuth token](https://api.slack.com/concepts/token-types), which must have the [`users:read` scope](https://api.slack.com/scopes/users:read) to access your Slack organization. **Type**: `string` ### [](#team_id)`team_id` The encoded ID of a Slack team by which to filter the list of returned users, which you can get from the [`team.info` Slack API method](https://api.slack.com/methods/team.info). If `team_id` is left empty, users from all teams within the organization are returned. **Type**: `string` **Default**: `""` --- # Page 95: slack **URL**: https://docs.redpanda.com/connect/components/inputs/slack.md --- # slack --- title: slack latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/slack page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/slack.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/slack.adoc description: Connects to Slack using Socket Mode and receives events, interactions, and slash commands. page-git-created-date: "2025-05-02" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Connects to Slack using [Socket Mode](https://api.slack.com/apis/socket-mode), and can receive events, interactions (automated and user-initiated), and slash commands. This input is useful for: - Building bots that can query or write data. - Sending events to data warehouses. You could also try pairing this input with Redpanda Connect’s AI processors, which use the prefixes `cohere`, `openai`, and `ollama`. Introduced in version 4.51.0. ```yml input: label: "" slack: app_token: "" # No default (required) bot_token: "" # No default (required) auto_replay_nacks: true ``` See also: [Examples](#examples) ## [](#metadata)Metadata Each message emitted from this input has an `@type` metadata flag to indicate the event type, either `"events_api"`, `"interactions"`, or `"slash_commands"`. ## [](#fields)Fields ### [](#app_token)`app_token` The app-level token, which starts with `xapp-`. The input uses it to open the Socket Mode connection, so it needs the [`connections:write` scope](https://api.slack.com/scopes/connections:write). **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` The OAuth token of your Slack bot user, which starts with `xoxb-`. **Type**: `string` ## [](#examples)Examples ### [](#echo-slackbot)Echo Slackbot A slackbot that echo messages from other users ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` --- # Page 96: socket_server **URL**: https://docs.redpanda.com/connect/components/inputs/socket_server.md --- # socket_server --- title: socket_server latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/socket_server page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/socket_server.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/socket_server.adoc description: Creates a server that receives a stream of messages over a TCP, UDP or Unix socket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Creates a server that receives a stream of messages over a TCP, UDP or Unix socket. #### Common ```yml input: label: "" socket_server: network: "" # No default (required) address: "" # No default (required) address_cache: "" # No default (optional) tls: cert_file: "" # No default (optional) key_file: "" # No default (optional) self_signed: false client_auth: no auto_replay_nacks: true scanner: lines: {} ``` #### Advanced ```yml input: label: "" socket_server: network: "" # No default (required) address: "" # No default (required) address_cache: "" # No default (optional) tcp: reuse_addr: false reuse_port: false tls: cert_file: "" # No default (optional) key_file: "" # No default (optional) self_signed: false client_auth: no auto_replay_nacks: true scanner: lines: {} ``` ## [](#fields)Fields ### [](#address)`address` The address to listen from. **Type**: `string` ```yaml # Examples: address: /tmp/benthos.sock # --- address: 0.0.0.0:6000 ``` ### [](#address_cache)`address_cache` An optional [`cache`](https://docs.redpanda.com/connect/components/caches/about/) within which this input should write its bound address once known. The key of the cache item containing the address will be the label of the component suffixed with `_address` (for example, `foo_address`), or `socket_server_address` when a label has not been provided. This is useful in situations where the address is dynamically allocated by the server (`127.0.0.1:0`) and you want to store the allocated address somewhere for reference by other systems and components. Requires version 4.25.0 or later. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#network)`network` A network type to accept. **Type**: `string` **Options**: `unix`, `tcp`, `udp`, `tls`, `unixgram` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS specific configuration, valid when the `network` is set to `tls`. Requires version 4.12.0 or later. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` PEM encoded certificate for use with TLS. **Type**: `string` ### [](#tls-client_auth)`tls.client_auth` Specifies how client authentication is handled when using TLS. Requires version 4.54.0 or later. **Type**: `string` **Default**: `no` | Option | Summary | | --- | --- | | no | client certificate is not requested nor required. | | request | will request client certificate, not require it. | | require_any | will accept any client certificate, even if not valid. | | require_valid | requires a valid client certificate. | | verify_if_given | will verify a certificate, if one is sent by the client. | ### [](#tls-key_file)`tls.key_file` PEM encoded private key for use with TLS. **Type**: `string` ### [](#tls-self_signed)`tls.self_signed` Whether to generate self signed certificates. **Type**: `bool` **Default**: `false` --- # Page 97: socket **URL**: https://docs.redpanda.com/connect/components/inputs/socket.md --- # socket --- title: socket latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/socket page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/socket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/socket.adoc description: Connects to a tcp or unix socket and consumes a continuous stream of messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Connects to a tcp or unix socket and consumes a continuous stream of messages. #### Common ```yml input: label: "" socket: network: "" # No default (required) address: "" # No default (required) auto_replay_nacks: true open_message_mapping: "" # No default (optional) scanner: lines: {} ``` #### Advanced ```yml input: label: "" socket: network: "" # No default (required) address: "" # No default (required) auto_replay_nacks: true open_message_mapping: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] scanner: lines: {} ``` ## [](#fields)Fields ### [](#address)`address` The address to connect to. **Type**: `string` ```yaml # Examples: address: /tmp/benthos.sock # --- address: 127.0.0.1:6000 ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#network)`network` A network type to assume (unix|tcp). **Type**: `string` **Options**: `unix`, `tcp` ### [](#open_message_mapping)`open_message_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to a string which will be sent upstream before the downstream data flow starts. Requires version 4.56.0 or later. **Type**: `string` ```yaml # Examples: open_message_mapping: root = "username,password" ``` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 98: spicedb_watch **URL**: https://docs.redpanda.com/connect/components/inputs/spicedb_watch.md --- # spicedb_watch --- title: spicedb_watch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/spicedb_watch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/spicedb_watch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/spicedb_watch.adoc description: Consume messages from the Watch API from SpiceDB. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-10" --- Consumes messages from the [Watch API](https://buf.build/authzed/api/docs/main:authzed.api.v1#authzed.api.v1.WatchService.Watch) of a [SpiceDB](https://authzed.com/docs/spicedb/getting-started/discovering-spicedb) instance. This input is useful if you have downstream applications that need to react to real-time changes in data managed by SpiceDB. Introduced in version 4.39.0. #### Common ```yml input: label: "" spicedb_watch: endpoint: "" # No default (required) bearer_token: "" cache: "" # No default (required) ``` #### Advanced ```yml input: label: "" spicedb_watch: endpoint: "" # No default (required) bearer_token: "" max_receive_message_bytes: 4MB cache: "" # No default (required) cache_key: authzed.com/spicedb/watch/last_zed_token tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` ## [](#authentication)Authentication For this input to authenticate with your SpiceDB instance, you must provide: - The [`endpoint`](#endpoint) of the SpiceDB instance - A [bearer token](#bearer_token) ## [](#configure-a-cache)Configure a cache You must use a cache resource to store the [ZedToken](https://authzed.com/docs/spicedb/concepts/consistency#zedtokens) (ID) of the latest message consumed and acknowledged by this input. Ideally, the cache should persist across restarts. This means that every time the input is initialized, it starts reading from the newest data updates. The following example uses a [`redis` cache](https://docs.redpanda.com/connect/components/rate_limits/redis/). ```yml # Example input: label: "" spicedb_watch: endpoint: grpc.authzed.com:443 bearer_token: "" cache: "spicedb_cache" cache_resources: - label: "spicedb_cache" redis: url: redis://:6379 ``` To learn more about cache configuration, see [Resources](https://docs.redpanda.com/connect/configuration/resources/) and the [Caches section](https://docs.redpanda.com/connect/components/caches/about/), which includes a range of cache components. ## [](#fields)Fields ### [](#bearer_token)`bearer_token` The SpiceDB bearer token to use to authenticate with your SpiceDB instance. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: bearer_token: t_your_token_here_1234567deadbeef ``` ### [](#cache)`cache` The cache resource that stores the ZedToken (ID) of the last message the input processed. When the input starts, it resumes the watch from the stored ZedToken, so it backfills updates it hasn’t read yet. The input stores the ZedToken when each message is acknowledged, whether or not the message reached its outputs, so a message that fails isn’t read again after a restart. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier to use when storing the ZedToken (ID) of the last message received. **Type**: `string` **Default**: `authzed.com/spicedb/watch/last_zed_token` ### [](#endpoint)`endpoint` The endpoint of your SpiceDB instance. **Type**: `string` ```yaml # Examples: endpoint: grpc.authzed.com:443 ``` ### [](#max_receive_message_bytes)`max_receive_message_bytes` The maximum message size (in bytes) this input can receive. If a message exceeds this limit, an `rpc error` is written to the Redpanda Connect logs. **Type**: `string` **Default**: `4MB` ```yaml # Examples: max_receive_message_bytes: 100MB # --- max_receive_message_bytes: 50mib ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#store-the-zedtoken-in-redis)Store the ZedToken in Redis Consumes updates from the Watch API and stores the ZedToken of the last acknowledged update in a Redis cache, so that the input resumes from that point after a restart. ```yaml input: spicedb_watch: endpoint: grpc.authzed.com:443 bearer_token: "${SPICEDB_TOKEN}" tls: enabled: true cache: spicedb_cache cache_resources: - label: spicedb_cache redis: url: redis://localhost:6379 ``` --- # Page 99: splunk **URL**: https://docs.redpanda.com/connect/components/inputs/splunk.md --- # splunk --- title: splunk latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/splunk page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/splunk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/splunk.adoc description: Consumes messages from Splunk. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Consumes messages from Splunk. Introduced in version 4.30.0. #### Common ```yml input: label: "" splunk: url: "" # No default (required) user: "" # No default (required) password: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" splunk: url: "" # No default (required) user: "" # No default (required) password: "" # No default (required) query: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#password)`password` Splunk account password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#query)`query` Splunk search query. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. Requires version 4.31.0 or later. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` Full HTTP Search API endpoint URL. **Type**: `string` ```yaml # Examples: url: https://foobar.splunkcloud.com/services/search/v2/jobs/export ``` ### [](#user)`user` Splunk account user. **Type**: `string` --- # Page 100: sql_raw **URL**: https://docs.redpanda.com/connect/components/inputs/sql_raw.md --- # sql_raw --- title: sql_raw latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/sql_raw page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sql_raw.adoc description: Executes a select query and creates a message for each row received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a select query and creates a message for each row received. Introduced in version 4.10.0. Once the rows from the query are exhausted this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to execute). #### Common ```yml input: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) ``` When the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate or for the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to execute. ## [](#examples)Examples ### [](#consumes-an-sql-table-using-a-query-as-an-input)Consumes an SQL table using a query as an input. This example performs an aggregate query that counts the rows for each name in a table, including only rows that were last updated more than 3600 seconds ago. ```yaml input: sql_raw: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable query: "SELECT name, count(*) FROM person WHERE last_updated < $1 GROUP BY name;" args_mapping: | root = [ now().ts_unix() - 3600 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: SELECT * FROM footable WHERE user_id = $1; ``` --- # Page 101: sql_select **URL**: https://docs.redpanda.com/connect/components/inputs/sql_select.md --- # sql_select --- title: sql_select latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/sql_select page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sql_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sql_select.adoc description: Executes a select query and creates a message for each row received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a select query and creates a message for each row received. Introduced in version 3.59.0. Once the rows from the query are exhausted this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/connect/components/inputs/sequence/) to execute). #### Common ```yml input: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) auto_replay_nacks: true init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) ``` ## [](#examples)Examples ### [](#consume-a-table-postgresql)Consume a Table (PostgreSQL) Here we define a pipeline that will consume all rows from a table created within the last hour by comparing the unix timestamp stored in the row column "created\_at": ```yaml input: sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] where: created_at >= ? args_mapping: | root = [ now().ts_unix() - 3600 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that evaluates to an array of values, one for each placeholder in the `where` field, in order. The values are passed to the driver as query arguments. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#columns)`columns[]` A list of columns to select. **Type**: `array` ```yaml # Examples: columns: - "*" # --- columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#prefix)`prefix` Optional SQL text to add before the `SELECT` keyword, such as a `WITH` clause. **Type**: `string` ### [](#suffix)`suffix` Optional SQL text to append after the generated `SELECT` statement, such as an `ORDER BY`, `LIMIT`, or locking clause. **Type**: `string` ### [](#table)`table` The table to select from. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Always use question marks for placeholders. They are converted to dollar syntax (`$1`) when the `postgres`, `pgx`, or `clickhouse` driver is used, and to colon syntax (`:1`) when the `oracle` or `gocosmos` driver is used. **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 102: stdin **URL**: https://docs.redpanda.com/connect/components/inputs/stdin.md --- # stdin --- title: stdin latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/stdin page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/stdin.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/stdin.adoc description: Consumes data piped to stdin, chopping it into individual messages according to the specified scanner. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes data piped to stdin, chopping it into individual messages according to the specified scanner. #### Common ```yml input: label: "" stdin: scanner: lines: {} auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" stdin: scanner: lines: {} auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` --- # Page 103: subprocess **URL**: https://docs.redpanda.com/connect/components/inputs/subprocess.md --- # subprocess --- title: subprocess latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/subprocess page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/subprocess.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/subprocess.adoc description: Executes a command, runs it as a subprocess, and consumes messages from it over stdout. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is in beta. Its behavior might change in a future release. Executes a command, runs it as a subprocess, and consumes messages from it over stdout. Messages are consumed according to a specified codec. The command is executed once and if it terminates the input also closes down gracefully. Alternatively, the field `restart_on_close` can be set to `true` in order to have Redpanda Connect re-execute the command each time it stops. The field `max_buffer` defines the maximum message size able to be read from the subprocess. This value should be set significantly above the real expected maximum message size. The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory. #### Common ```yml input: label: "" subprocess: name: "" # No default (required) args: [] codec: lines restart_on_exit: false ``` #### Advanced ```yml input: label: "" subprocess: name: "" # No default (required) args: [] codec: lines restart_on_exit: false max_buffer: 65536 ``` ## [](#fields)Fields ### [](#args)`args[]` A list of arguments to provide the command. **Type**: `array` **Default**: `[]` ### [](#codec)`codec` The way in which messages should be consumed from the subprocess. **Type**: `string` **Default**: `lines` **Options**: `lines` ### [](#max_buffer)`max_buffer` The maximum expected size of an individual message. **Type**: `int` **Default**: `65536` ### [](#name)`name` The command to execute as a subprocess. **Type**: `string` ```yaml # Examples: name: cat # --- name: sed # --- name: awk ``` ### [](#restart_on_exit)`restart_on_exit` Whether the command should be re-executed each time the subprocess ends. **Type**: `bool` **Default**: `false` --- # Page 104: tigerbeetle_cdc **URL**: https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc.md --- # tigerbeetle_cdc --- title: tigerbeetle_cdc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/tigerbeetle_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/tigerbeetle_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/tigerbeetle_cdc.adoc description: Enables TigerBeetle CDC streaming for Redpanda Connect. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-10-10" --- Introduced in version 4.65.0 Enables integration with [TigerBeetle’s Change Data Capture](https://docs.tigerbeetle.com/operating/cdc/) (CDC) streaming for Redpanda Connect. This input component connects to a TigerBeetle cluster and continuously listens for real-time changes (such as account updates, transfers, and other ledger events). Each detected change is converted into a message and ingested into Redpanda Connect for further processing. Use this input when you need to synchronize TigerBeetle’s financial ledger events with downstream systems, enable event-driven architectures, or build real-time analytics pipelines. Typical use cases include auditing, monitoring, fraud detection, and maintaining up-to-date replicas of financial data in other services. #### Common ```yml input: label: "" tigerbeetle_cdc: cluster_id: "" # No default (required) addresses: [] # No default (required) progress_cache: "" # No default (required) rate_limit: "" event_count_max: 2730 idle_interval_ms: 1000 timestamp_initial: "" timeout_seconds: 15 auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" tigerbeetle_cdc: cluster_id: "" # No default (required) addresses: [] # No default (required) progress_cache: "" # No default (required) rate_limit: "" event_count_max: 2730 idle_interval_ms: 1000 timestamp_initial: "" timeout_seconds: 15 auto_replay_nacks: true ``` Each message produced by this input is a JSON object representing a TigerBeetle CDC event. The structure includes details about the event type, ledger, transfer, debit and credit accounts, and associated timestamps. For example: ```json { "timestamp": "1745328372758695656", "type": "single_phase", "ledger": 2, "transfer": { "id": "9082709", "amount": "3794", "pending_id": "0", "user_data_128": "79248595801719937611592367840129079151", "user_data_64": "13615171707598273871", "user_data_32": 3229992513, "timeout": 0, "code": 20295, "flags": 0, "timestamp": "1745328372758695656" }, "debit_account": { "id": "3750", "debits_pending": "0", "debits_posted": "8463768", "credits_pending": "0", "credits_posted": "8861179", "user_data_128": "118966247877720884212341541320399553321", "user_data_64": "526432537153007844", "user_data_32": 4157247332, "code": 1, "flags": 0, "timestamp": "1745328270103398016" }, "credit_account": { "id": "6765", "debits_pending": "0", "debits_posted": "8669204", "credits_pending": "0", "credits_posted": "8637251", "user_data_128": "43670023860556310170878798978091998141", "user_data_64": "12485093662256535374", "user_data_32": 1924162092, "code": 1, "flags": 0, "timestamp": "1745328270103401031" } } ``` ## [](#requirements)Requirements Before using the TigerBeetle CDC input, ensure that your environment meets the following requirements: - TigerBeetle cluster version 0.16.57 or later is required. - The TigerBeetle client version must not be newer than the cluster version to ensure compatibility. - Consumers must implement idempotency when processing messages to handle potential duplicates. > 📝 **NOTE** > > This component is available only in builds of Redpanda Connect that enable cgo. The standard release binaries, the Docker images, and the standard `rpk connect` plugin don’t include it. To use it, do one of the following: > > - Download the `redpanda-connect-cgo__linux_amd64` archive from the [GitHub releases page](https://github.com/redpanda-data/connect/releases). This archive is available for Linux AMD64 only. > > - Build Redpanda Connect from source with `CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata ./cmd/redpanda-connect`. The build needs a C compiler. > > > The cgo archive and the source build both need the ZeroMQ library (`libzmq`), because they include the `zmq4` components. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `event_type`: One of "single\_phase", "two\_phase\_pending", "two\_phase\_posted", "two\_phase\_voided", or "two\_phase\_expired". - `ledger`: The ledger code. - `transfer_code`: The transfer code. - `debit_account_code`: The debit account code. - `credit_account_code`: The credit account code. - `timestamp`: The unique event timestamp with nanosecond resolution. - `timestamp_ms`: The event timestamp with millisecond resolution. ## [](#guarantees)Guarantees This input guarantees at-least-once semantics, and makes a best effort to prevent duplicate messages. However, during crash recovery, it may replay unacknowledged messages that could have been already delivered to consumers. It is the consumer’s responsibility to perform idempotency checks when processing messages. ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of IP addresses of all the TigerBeetle replicas in the cluster. The order of addresses must correspond to the order of replicas. **Type**: `array` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#cluster_id)`cluster_id` The TigerBeetle unique 128-bit cluster ID. **Type**: `string` ### [](#event_count_max)`event_count_max` The maximum number of events fetched from TigerBeetle per **request**. Must be greater than zero. **Type**: `int` **Default**: `2730` ### [](#idle_interval_ms)`idle_interval_ms` The time interval in milliseconds to wait before querying again when the last request returned no events. Must be greater than zero. **Type**: `int` **Default**: `1000` ### [](#progress_cache)`progress_cache` A [cache resource](https://docs.redpanda.com/connect/components/caches/about/) used to track progress by storing the last acknowledged timestamp. This allows Redpanda Connect to resume from the latest delivered event upon restart. **Type**: `string` ### [](#rate_limit)`rate_limit` An optional [rate limit](https://docs.redpanda.com/connect/components/rate_limits/about/) to throttle the number of **requests** made to TigerBeetle. **Type**: `string` **Default**: `""` ### [](#timeout_seconds)`timeout_seconds` The timeout in seconds, for querying the TigerBeetle cluster. Requires version 4.82.0 or later. **Type**: `int` **Default**: `15` ### [](#timestamp_initial)`timestamp_initial` The initial timestamp to start extracting events from. If not defined, all events since the beginning will be included. Ignored if a more recent timestamp has already been acknowledged. This is a TigerBeetle timestamp with nanosecond precision. **Type**: `string` **Default**: `""` --- # Page 105: timeplus **URL**: https://docs.redpanda.com/connect/components/inputs/timeplus.md --- # timeplus --- title: timeplus latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/timeplus page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/timeplus.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/timeplus.adoc description: Executes a query on Timeplus Enterprise and creates a message from each row received. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-10" --- Executes a streaming or table query on [Timeplus Enterprise (Cloud or Self-Hosted)](https://docs.timeplus.com/) or the `timeplusd` component, and creates a structured message for each table row received. If you execute a streaming query, this input runs until the query terminates. For table queries, it shuts down after all rows returned by the query are exhausted. ```yml input: label: "" timeplus: query: "" # No default (required) url: tcp://localhost:8463 workspace: "" # No default (optional) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) ``` ## [](#examples)Examples ### [](#from-timeplus-enterprise-cloud-via-http)From Timeplus Enterprise Cloud via HTTP Create an API key in the Timeplus Enterprise Cloud web console, then set it in the `apikey` field. ```yaml input: timeplus: url: https://us-west-2.timeplus.cloud workspace: my_workspace_id query: select * from iot apikey: ``` ### [](#from-timeplus-enterprise-self-hosted-via-http)From Timeplus Enterprise (self-hosted) via HTTP For self-hosted Timeplus Enterprise, specify the username and password as well as the URL of the app server. ```yaml input: timeplus: url: http://localhost:8000 workspace: my_workspace_id query: select * from iot username: username password: pw ``` ### [](#from-timeplus-enterprise-self-hosted-via-tcp)From Timeplus Enterprise (self-hosted) via TCP Make sure the scheme of the `url` is `tcp`. ```yaml input: timeplus: url: tcp://localhost:8463 query: select * from iot username: timeplus password: timeplus ``` ## [](#fields)Fields ### [](#apikey)`apikey` The API key for the Timeplus Enterprise REST API. You need to generate the key in the web console of Timeplus Enterprise (Cloud). This field is required if you are reading messages from Timeplus Enterprise (Cloud). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#password)`password` The password for the Timeplus application server. This field is required if you are reading messages from Timeplus Enterprise (Self-Hosted) or `timeplusd`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#query)`query` The query to execute on Timeplus Enterprise (Cloud or Self-Hosted) or `timeplusd`. **Type**: `string` ```yaml # Examples: query: select * from iot # --- query: select count(*) from table(iot) ``` ### [](#url)`url` The URL of your Timeplus instance, which should always include the schema and host. **Type**: `string` **Default**: `tcp://localhost:8463` ### [](#username)`username` The username for the Timeplus application server. This field is required if you are reading messages from Timeplus Enterprise (Self-Hosted) or `timeplusd`. **Type**: `string` ### [](#workspace)`workspace` The ID of the workspace you want to read messages from. This field is required if you are connecting to Timeplus Enterprise (Cloud or Self-Hosted) using HTTP. **Type**: `string` --- # Page 106: twitter_search **URL**: https://docs.redpanda.com/connect/components/inputs/twitter_search.md --- # twitter_search --- title: twitter_search latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/twitter_search page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/twitter_search.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/twitter_search.adoc description: Consumes tweets matching a given search using the Twitter recent search V2 API. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is experimental. It might change or be removed in a future release. Consumes tweets matching a given search using the Twitter recent search V2 API. Continuously polls the [Twitter recent search V2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent) for tweets that match a given search query. Each tweet received is emitted as a JSON object message, with a field `id` and `text` by default. Extra fields [can be obtained from the search API](https://developer.twitter.com/en/docs/twitter-api/fields) when listed with the `tweet_fields` field. In order to paginate requests that are made the ID of the latest received tweet is stored in a [cache resource](https://docs.redpanda.com/connect/components/caches/about/), which is then used by subsequent requests to ensure only tweets after it are consumed. It is recommended that the cache you use is persistent so that Redpanda Connect can resume searches at the correct place on a restart. Authentication is done using OAuth 2.0 credentials which can be generated within the [Twitter developer portal](https://developer.twitter.com). #### Common ```yml input: label: "" twitter_search: query: "" # No default (required) tweet_fields: [] poll_period: 1m backfill_period: 5m cache: "" # No default (required) api_key: "" # No default (required) api_secret: "" # No default (required) ``` #### Advanced ```yml input: label: "" twitter_search: query: "" # No default (required) tweet_fields: [] poll_period: 1m backfill_period: 5m cache: "" # No default (required) cache_key: last_tweet_id rate_limit: "" api_key: "" # No default (required) api_secret: "" # No default (required) ``` ## [](#fields)Fields ### [](#api_key)`api_key` An API key for OAuth 2.0 authentication. It is recommended that you populate this field using [environment variables](https://docs.redpanda.com/connect/configuration/interpolation/). **Type**: `string` ### [](#api_secret)`api_secret` An API secret for OAuth 2.0 authentication. It is recommended that you populate this field using [environment variables](https://docs.redpanda.com/connect/configuration/interpolation/). **Type**: `string` ### [](#backfill_period)`backfill_period` A duration string indicating the maximum age of tweets to acquire when starting a search. **Type**: `string` **Default**: `5m` ### [](#cache)`cache` A cache resource to use for request pagination. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier used when storing the ID of the last tweet received. **Type**: `string` **Default**: `last_tweet_id` ### [](#poll_period)`poll_period` The length of time (as a duration string) to wait between each search request. This field can be set empty, in which case requests are made at the limit set by the rate limit. This field also supports cron expressions. **Type**: `string` **Default**: `1m` ### [](#query)`query` A search expression to use. **Type**: `string` ### [](#rate_limit)`rate_limit` An optional rate limit resource to restrict API requests with. **Type**: `string` **Default**: `""` ### [](#tweet_fields)`tweet_fields[]` An optional list of additional fields to obtain for each tweet, by default only the fields `id` and `text` are returned. For more info refer to the [twitter API docs](https://developer.twitter.com/en/docs/twitter-api/fields). **Type**: `array` **Default**: `[]` --- # Page 107: websocket **URL**: https://docs.redpanda.com/connect/components/inputs/websocket.md --- # websocket --- title: websocket latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/websocket page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/websocket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/websocket.adoc description: Connects to a websocket server and continuously receives messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Connects to a websocket server and continuously receives messages. It is possible to configure an `open_message`, which when set to a non-empty string will be sent to the websocket server each time a connection is first established. #### Common ```yml input: label: "" websocket: url: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml input: label: "" websocket: url: "" # No default (required) proxy_url: "" # No default (optional) open_message: "" # No default (optional) open_message_type: binary max_message_size: 33554432 auto_replay_nacks: true tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] connection: max_retries: 0 # No default (optional) oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. Requires version 4.27.0 or later. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#connection)`connection` Customise how websocket connection attempts are made. Requires version 4.16.0 or later. **Type**: `object` ### [](#connection-max_retries)`connection.max_retries` An optional limit to the number of consecutive retry attempts that will be made before abandoning the connection altogether and gracefully terminating the input. When all inputs terminate in this way the service (or stream) will shut down. If set to zero connections will never be reattempted upon a failure. If set below zero this field is ignored (effectively unset). **Type**: `int` ```yaml # Examples: max_retries: -1 # --- max_retries: 10 ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_message_size)`max_message_size` A maximum size in bytes for individual messages received from the server. When a message exceeding this limit is received the connection is closed with a 1009 (message too big) status and the input reconnects. A value of 0 disables the limit, which allows the server to make this process allocate an unbounded amount of memory and is therefore not recommended. Requires version 4.106.0 or later. **Type**: `int` **Default**: `33554432` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#open_message)`open_message` An optional message to send to the server upon connection. **Type**: `string` ### [](#open_message_type)`open_message_type` An optional flag to indicate the data type of open\_message. Requires version 4.12.0 or later. **Type**: `string` **Default**: `binary` | Option | Summary | | --- | --- | | binary | Binary data open_message. | | text | Text data open_message. The text message payload is interpreted as UTF-8 encoded text data. | ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. Requires version 4.30.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. **Type**: `string` ```yaml # Examples: url: ws://localhost:4195/get/ws ``` --- # Page 108: zmq4 **URL**: https://docs.redpanda.com/connect/components/inputs/zmq4.md --- # zmq4 --- title: zmq4 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: inputs/zmq4 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/zmq4.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/zmq4.adoc description: Consumes messages from a ZeroMQ socket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes messages from a ZeroMQ socket. #### Common ```yml input: label: "" zmq4: urls: [] # No default (required) bind: false socket_type: "" # No default (required) sub_filters: [] ``` #### Advanced ```yml input: label: "" zmq4: urls: [] # No default (required) bind: false socket_type: "" # No default (required) sub_filters: [] high_water_mark: 0 poll_timeout: 5s ``` > 📝 **NOTE** > > This component is available only in builds of Redpanda Connect that enable cgo. The standard release binaries, the Docker images, and the standard `rpk connect` plugin don’t include it. To use it, do one of the following: > > - Download the `redpanda-connect-cgo__linux_amd64` archive from the [GitHub releases page](https://github.com/redpanda-data/connect/releases). This archive is available for Linux AMD64 only. > > - Build Redpanda Connect from source with `CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata ./cmd/redpanda-connect`. The build needs a C compiler. > > > The cgo archive and the source build both need the ZeroMQ library (`libzmq`), because they include the `zmq4` components. ## [](#fields)Fields ### [](#bind)`bind` Whether to bind to the URLs and wait for peers to connect, instead of connecting to them. **Type**: `bool` **Default**: `false` ### [](#high_water_mark)`high_water_mark` The message high water mark to use. **Type**: `int` **Default**: `0` ### [](#poll_timeout)`poll_timeout` The poll timeout to use. **Type**: `string` **Default**: `5s` ### [](#socket_type)`socket_type` The socket type to connect as. **Type**: `string` **Options**: `PULL`, `SUB` ### [](#sub_filters)`sub_filters[]` The topics that the socket subscribes to when `socket_type` is `SUB`. ZeroMQ delivers a message only when its first frame begins with one of these prefixes, so specifying a single sub\_filter of `''` subscribes to everything. At least one filter is required with a SUB socket. **Type**: `array` **Default**: `[]` ### [](#urls)`urls[]` A list of ZeroMQ endpoints to connect to, or to bind when `bind` is `true`, using a transport such as `tcp://`, `ipc://`, or `inproc://`. If an item in the list contains commas, it is split into multiple endpoints. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:5555" ``` --- # Page 109: Logger **URL**: https://docs.redpanda.com/connect/components/logger/about.md --- # Logger --- title: Logger latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: logger/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: logger/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/logger/about.adoc description: Learn how to configure Redpanda Connect logging, including log formats and levels. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Redpanda Connect logging prints to stdout (or stderr if your output is stdout) and is formatted as [logfmt](https://brandur.org/logfmt) by default. Use these configuration options to change both the logging formats as well as the destination of logs. > 💡 **TIP** > > You can also configure the [`redpanda` component](https://docs.redpanda.com/connect/components/redpanda/about/) to send logs and status updates to topics on a Redpanda cluster. For monitoring pipeline health with status events, see [Monitor Pipeline Status](https://docs.redpanda.com/connect/guides/monitor-pipeline-status/). #### Common ```yml logger: level: INFO format: logfmt add_timestamp: true static_fields: "@service": "redpanda-connect" ``` #### Advanced ```yml logger: level: INFO format: logfmt add_timestamp: true level_name: level timestamp_name: time message_name: msg static_fields: "@service": "redpanda-connect" file: path: "" rotate: false rotate_max_age_days: 0 ``` ## [](#fields)Fields ### [](#add_timestamp)`add_timestamp` Whether to include timestamps in logs. **Type**: `bool` **Default**: `true` ### [](#file)`file` Experimental: Specify fields for optionally writing logs to a file. **Type**: `object` ### [](#file-path)`file.path` The file path to write logs to, if the file does not exist it will be created. Leave this field empty or unset to disable file based logging. **Type**: `string` **Default**: `""` ### [](#file-rotate)`file.rotate` Whether to rotate log files automatically. **Type**: `bool` **Default**: `false` ### [](#file-rotate_max_age_days)`file.rotate_max_age_days` The maximum number of days to retain old log files based on the timestamp encoded in their filename, after which they are deleted. Setting to zero disables this mechanism. **Type**: `int` **Default**: `0` ### [](#format)`format` Set the format of emitted logs. **Type**: `string` **Default**: `logfmt` **Options**: `json`, `logfmt` ### [](#level)`level` Set the minimum severity level for emitting logs. **Type**: `string` **Default**: `INFO` **Options**: `OFF`, `FATAL`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`, `ALL`, `NONE` ### [](#level_name)`level_name` The name of the level field added to logs when the `format` is `json`. **Type**: `string` **Default**: `level` ### [](#message_name)`message_name` The name of the message field added to logs when the `format` is `json`. **Type**: `string` **Default**: `msg` ### [](#static_fields)`static_fields` A map of key/value pairs to add to each structured log. **Type**: `object` **Default**: ```yaml "@service": "redpanda-connect" ``` ### [](#timestamp_name)`timestamp_name` The name of the timestamp field added to logs when `add_timestamp` is set to `true` and the `format` is `json`. **Type**: `string` **Default**: `time` --- # Page 110: Metrics **URL**: https://docs.redpanda.com/connect/components/metrics/about.md --- # Metrics --- title: Metrics latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/about.adoc description: Learn how Redpanda Connect emits metrics about pipeline components, and how to route them to a destination such as Prometheus. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Redpanda Connect emits lots of metrics in order to expose how components configured within your pipeline are behaving. You can configure exactly where these metrics end up with the config field `metrics`, which describes a metrics format and destination. For example, if you wished to push them via the StatsD protocol you could use this configuration: ```yaml metrics: statsd: address: localhost:8125 flush_period: 100ms ``` The default metrics configuration is to expose Prometheus metrics on the [service-wide HTTP endpoint](https://docs.redpanda.com/connect/components/http/about/) at the endpoints `/metrics` and `/stats`. ## [](#timings)Timings It’s worth noting that timing metrics within Redpanda Connect are measured in nanoseconds and are therefore named with a `_ns` suffix. However, some exporters do not support this level of precision and are downgraded, or have the unit converted for convenience. In these cases the exporter documentation outlines the conversion and why it is made. ## [](#metric-names)Metric names Each major Redpanda Connect component type emits one or more metrics with the name prefixed by the type. These metrics are intended to provide an overview of behavior, performance and health. Some specific component implementations may provide their own unique metrics on top of these standardized ones, these extra metrics can be found listed on their respective documentation pages. ## [](#inputs)Inputs - `input_received`: A count of the number of messages received by the input. - `input_latency_ns`: Measures the roundtrip latency in nanoseconds from the point at which a message is read up to the moment the message has either been acknowledged by an output, has been stored within a buffer, or has been rejected (nacked). - `batch_created`: A count of each time an input-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. - `input_connection_up`: For continuous stream based inputs represents a count of the number of the times the input has successfully established a connection to the target source. For poll based inputs that do not retain an active connection this value will increment once. - `input_connection_failed`: For continuous stream based inputs represents a count of the number of times the input has failed to establish a connection to the target source. - `input_connection_lost`: For continuous stream based inputs represents a count of the number of times the input has lost a previously established connection to the target source. > ⚠️ **CAUTION** > > The behavior of connection metrics may differ based on input type due to certain libraries and protocols obfuscating the concept of a single connection. ### [](#buffers)Buffers - `buffer_received`: A count of the number of messages written to the buffer. - `buffer_batch_received`: A count of the number of message batches written to the buffer. - `buffer_sent`: A count of the number of messages read from the buffer. - `buffer_batch_sent`: A count of the number of message batches read from the buffer. - `buffer_latency_ns`: Measures the roundtrip latency in nanoseconds from the point at which a message is read from the buffer up to the moment it has been acknowledged by the output. - `batch_created`: A count of each time a buffer-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. ### [](#processors)Processors - `processor_received`: A count of the number of messages the processor has been executed upon. - `processor_batch_received`: A count of the number of message batches the processor has been executed upon. - `processor_sent`: A count of the number of messages the processor has returned. - `processor_batch_sent`: A count of the number of message batches the processor has returned. - `processor_error`: A count of the number of times the processor has errored. In cases where an error is batch-wide the count is incremented by one, and therefore would not match the number of messages. - `processor_latency_ns`: Latency of message processing in nanoseconds. When a processor acts upon a batch of messages this latency measures the time taken to process all messages of the batch. ### [](#outputs)Outputs - `output_sent`: A count of the number of messages sent by the output. - `output_batch_sent`: A count of the number of message batches sent by the output. - `output_error`: A count of the number of send attempts that have failed. On failed batched sends this count is incremented once only. - `output_latency_ns`: Latency of writes in nanoseconds. This metric may not be populated by outputs that are pull-based such as the `http_server`. - `batch_created`: A count of each time an output-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. - `output_connection_up`: For continuous stream based outputs represents a count of the number of the times the output has successfully established a connection to the target sink. For poll based outputs that do not retain an active connection this value will increment once. - `output_connection_failed`: For continuous stream based outputs represents a count of the number of times the output has failed to establish a connection to the target sink. - `output_connection_lost`: For continuous stream based outputs represents a count of the number of times the output has lost a previously established connection to the target sink. > ⚠️ **CAUTION** > > The behavior of connection metrics may differ based on output type due to certain libraries and protocols obfuscating the concept of a single connection. ### [](#caches)Caches All cache metrics have a label `operation` denoting the operation that triggered the metric series, one of; `add`, `get`, `set` or `delete`. - `cache_success`: A count of the number of successful cache operations. - `cache_error`: A count of the number of cache operations that resulted in an error. - `cache_latency_ns`: Latency of operations in nanoseconds. - `cache_not_found`: A count of the number of get operations that yielded no value due to the item not being found. This count is separate from `cache_error`. - `cache_duplicate`: A count of the number of add operations that were aborted due to the key already existing. This count is separate from `cache_error`. ### [](#rate-limits)Rate limits - `rate_limit_checked`: A count of the number of times the rate limit has been probed. - `rate_limit_triggered`: A count of the number of times the rate limit has been triggered by a probe. - `rate_limit_error`: A count of the number of times the rate limit has errored when probed. ## [](#metric-labels)Metric labels The standard metric names are unique to the component type, but a benthos config may consist of any number of component instantiations. In order to provide a metrics series that is unique for each instantiation Redpanda Connect adds labels (or tags) that uniquely identify the instantiation. These labels are as follows: ### [](#path)`path` The `path` label contains a string representation of the position of a component instantiation within a config in a format that would locate it within a Bloblang mapping, beginning at `root`. This path is a best attempt and may not exactly represent the source component position in all cases and is intended to be used for assisting observability only. This is the highest cardinality label since paths will change as configs are updated and expanded. It is therefore worth removing this label with a [mapping](#metric-mapping) in cases where you wish to restrict the number of unique metric series. ### [](#label)`label` The `label` label contains the unique label configured for a component emitting the metric series, or is empty for components that do not have a configured label. This is the most useful label for uniquely identifying a series for a component. ### [](#stream)`stream` The `stream` label is present in a metric series emitted from a stream config executed when Redpanda Connect is running in [streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/), and is populated with the stream name. ## [](#example)Example The following Redpanda Connect configuration: ```yaml input: label: foo http_server: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() output: label: bar stdout: {} metrics: prometheus: {} ``` Would produce the following metrics series: ```text input_latency_ns{label="foo",path="root.input"} input_received{endpoint="post",label="foo",path="root.input"} input_received{endpoint="websocket",label="foo",path="root.input"} processor_batch_received{label="",path="root.pipeline.processors.0"} processor_batch_sent{label="",path="root.pipeline.processors.0"} processor_error{label="",path="root.pipeline.processors.0"} processor_latency_ns{label="",path="root.pipeline.processors.0"} processor_received{label="",path="root.pipeline.processors.0"} processor_sent{label="",path="root.pipeline.processors.0"} output_batch_sent{label="bar",path="root.output"} output_connection_failed{label="bar",path="root.output"} output_connection_lost{label="bar",path="root.output"} output_connection_up{label="bar",path="root.output"} output_error{label="bar",path="root.output"} output_latency_ns{label="bar",path="root.output"} output_sent{label="bar",path="root.output"} ``` ## [](#metric-mapping)Metric mapping Since Redpanda Connect emits a large variety of metrics it is often useful to restrict or modify the metrics that are emitted. This can be done using the [Bloblang mapping language](https://docs.redpanda.com/connect/guides/bloblang/about/) in the field `metrics.mapping`. This is a mapping executed for each metric that is registered within the Redpanda Connect service and allows you to delete an entire series, modify the series name and delete or modify individual labels. Within the mapping the input document (referenced by the keyword `this`) is a string value containing the metric name, and the resulting document (referenced by the keyword `root`) must be a string value containing the resulting name. As is standard in Bloblang mappings, if the value of `root` is not assigned within the mapping then the metric name remains unchanged. If the value of `root` is `deleted()` then the metric series is dropped. Labels can be referenced as metadata values with the function `meta`, where if the label does not exist in the series being mapped the value `null` is returned. Labels can be changed by using meta assignments, and can be assigned `deleted()` in order to remove them. For example, the following mapping removes all but the `label` label entirely, which reduces the cardinality of each series. It also renames the `label` (for some reason) so that labels containing meows now contain woofs. Finally, the mapping restricts the metrics emitted to only three series; one for the input count, one for processor errors, and one for the output count, it does this by looking up metric names in a static array of allowed names, and if not present the `root` is assigned `deleted()`: ```yaml metrics: mapping: | # Delete all pre-existing labels meta = deleted() # Re-add the `label` label with meows replaced with woofs meta label = meta("label").replace("meow", "woof") # Delete all metric series that aren't in our list root = if ![ "input_received", "processor_error", "output_sent", ].contains(this) { deleted() } prometheus: use_histogram_timing: false ``` --- # Page 111: aws_cloudwatch **URL**: https://docs.redpanda.com/connect/components/metrics/aws_cloudwatch.md --- # aws_cloudwatch --- title: aws_cloudwatch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/aws_cloudwatch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/aws_cloudwatch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/aws_cloudwatch.adoc description: Send metrics to AWS CloudWatch using the PutMetricData endpoint. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Send metrics to AWS CloudWatch using the PutMetricData endpoint. Introduced in version 3.36.0. ## [](#timing-metrics)Timing metrics The smallest timing unit that CloudWatch supports is microseconds, therefore timing metrics are automatically downgraded to microseconds (by dividing delta values by 1000). This conversion will also apply to custom timing metrics produced with a `metric` processor. ## [](#billing)Billing AWS bills per metric series exported, it is therefore STRONGLY recommended that you reduce the metrics that are exposed with a `mapping` like this: ```yaml metrics: mapping: | if ![ "input_received", "input_latency", "output_sent", ].contains(this) { deleted() } aws_cloudwatch: namespace: Foo ``` ### Common ```yml metrics: aws_cloudwatch: namespace: Benthos ``` ### Advanced ```yml metrics: aws_cloudwatch: namespace: Benthos flush_period: 100ms region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#fields)Fields ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#flush_period)`flush_period` The period of time between PutMetricData requests. **Type**: `string` **Default**: `100ms` ### [](#namespace)`namespace` The namespace used to distinguish metrics from other services. **Type**: `string` **Default**: `Benthos` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 112: influxdb **URL**: https://docs.redpanda.com/connect/components/metrics/influxdb.md --- # influxdb --- title: influxdb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/influxdb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/influxdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/influxdb.adoc description: Send metrics to InfluxDB 1.x using the /write endpoint. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Send metrics to InfluxDB 1.x using the `/write` endpoint. Introduced in version 3.36.0. See [https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint](https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint) for further details on the write API. #### Common ```yml metrics: influxdb: url: "" # No default (required) db: "" # No default (required) ``` #### Advanced ```yml metrics: influxdb: url: "" # No default (required) db: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] username: "" password: "" include: runtime: "" debug_gc: "" interval: 1m ping_interval: 20s precision: s timeout: 5s tags: {} retention_policy: "" # No default (optional) write_consistency: "" # No default (optional) ``` ## [](#fields)Fields ### [](#db)`db` The name of the database to use. **Type**: `string` ### [](#include)`include` Optional additional metrics to collect, enabling these metrics may have some performance implications as it acquires a global semaphore and does `stoptheworld()`. **Type**: `object` ### [](#include-debug_gc)`include.debug_gc` A duration string indicating how often to poll and collect GC metrics. Leave empty to disable this metric. **Type**: `string` **Default**: `""` ```yaml # Examples: debug_gc: 1m ``` ### [](#include-runtime)`include.runtime` A duration string indicating how often to poll and collect runtime metrics. Leave empty to disable this metric **Type**: `string` **Default**: `""` ```yaml # Examples: runtime: 1m ``` ### [](#interval)`interval` A duration string indicating how often metrics should be flushed. **Type**: `string` **Default**: `1m` ### [](#password)`password` A password (when applicable). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#ping_interval)`ping_interval` A duration string indicating how often to ping the host. **Type**: `string` **Default**: `20s` ### [](#precision)`precision` \[ns|us|ms|s\] timestamp precision passed to write api. **Type**: `string` **Default**: `s` ### [](#retention_policy)`retention_policy` Sets the retention policy for each write. **Type**: `string` ### [](#tags)`tags` Global tags added to each metric. **Type**: `object` **Default**: `{}` ```yaml # Examples: tags: hostname: localhost zone: danger ``` ### [](#timeout)`timeout` How long to wait for response for both ping and writing metrics. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` A URL of the format `[https|http|udp]://host:port` to the InfluxDB host. **Type**: `string` ### [](#username)`username` A username (when applicable). **Type**: `string` **Default**: `""` ### [](#write_consistency)`write_consistency` \[any|one|quorum|all\] sets write consistency when available. **Type**: `string` --- # Page 113: json_api **URL**: https://docs.redpanda.com/connect/components/metrics/json_api.md --- # json_api --- title: json_api latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/json_api page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/json_api.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/json_api.adoc description: Serves metrics as JSON object with the service wide HTTP service at the endpoints /stats and /metrics. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Serves metrics as JSON object with the service wide HTTP service at the endpoints `/stats` and `/metrics`. This metrics type is useful for debugging as it provides a human readable format that you can parse with tools such as `jq` ```yml metrics: json_api: {} ``` --- # Page 114: logger **URL**: https://docs.redpanda.com/connect/components/metrics/logger.md --- # logger --- title: logger latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/logger page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/logger.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/logger.adoc description: Prints aggregated metrics through the logger. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Prints aggregated metrics through the logger. ```yml metrics: logger: push_interval: "" # No default (optional) flush_metrics: false ``` Prints each metric produced by Redpanda Connect as a log event (level `info` by default) during shutdown, and optionally on an interval. This metrics type is useful for debugging pipelines when you only have access to the logger output and not the service-wide server. Otherwise it’s recommended that you use either the `prometheus` or `json_api` types. ## [](#fields)Fields ### [](#flush_metrics)`flush_metrics` Whether counters and timing metrics should be reset to 0 each time metrics are printed. **Type**: `bool` **Default**: `false` ### [](#push_interval)`push_interval` An optional period of time to continuously print all metrics. **Type**: `string` --- # Page 115: none **URL**: https://docs.redpanda.com/connect/components/metrics/none.md --- # none --- title: none latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/none page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/none.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/none.adoc description: Disable metrics entirely. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Disable metrics entirely. ## Common ```yml metrics: none: {} ``` ## Advanced ```yml metrics: none: {} ``` --- # Page 116: open_telemetry_collector **URL**: https://docs.redpanda.com/connect/components/metrics/open_telemetry_collector.md --- # open_telemetry_collector --- title: open_telemetry_collector latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/open_telemetry_collector page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/open_telemetry_collector.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/open_telemetry_collector.adoc description: Send metrics to an Open Telemetry collector. page-git-created-date: "2026-04-20" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Send metrics to an [Open Telemetry collector](https://opentelemetry.io/docs/collector/). Exports Redpanda Connect metrics to one or more OpenTelemetry Collector endpoints over HTTP or gRPC for aggregation and onward export. Metrics are encoded using the OpenTelemetry Metrics protocol and can be sent to any collector endpoint that supports OTLP. You can configure multiple collector endpoints (both HTTP and gRPC simultaneously). All configured endpoints will receive the same metrics data. This is useful for redundancy or sending metrics to multiple observability platforms. ### Common ```yml metrics: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) ``` ### Advanced ```yml metrics: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) tags: {} ``` ## [](#fields)Fields ### [](#grpc)`grpc[]` A list of grpc collectors. **Type**: `array` ### [](#grpc-address)`grpc[].address` The endpoint of a collector to send events to. **Type**: `string` ```yaml # Examples: address: localhost:4317 ``` ### [](#grpc-secure)`grpc[].secure` Connect to the collector with client transport security **Type**: `bool` **Default**: `false` ### [](#http)`http[]` A list of http collectors. **Type**: `array` ### [](#http-address)`http[].address` The endpoint of a collector to send events to. **Type**: `string` ```yaml # Examples: address: localhost:4318 ``` ### [](#http-secure)`http[].secure` Connect to the collector over HTTPS **Type**: `bool` **Default**: `false` ### [](#service)`service` The name of the service in metrics. **Type**: `string` **Default**: `benthos` ### [](#tags)`tags` A map of tags to add to all exported spans and metrics. **Type**: `object` **Default**: `{}` ## [](#usage)Usage The most common setup uses a local OpenTelemetry Collector running as a sidecar or daemon, which then forwards metrics to your observability backend: ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: localhost:4317 ``` For production deployments with remote collectors, enable TLS: ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: otel-collector.example.com:4317 secure: true ``` Use the `tags` field to add labels to all exported metrics for filtering and grouping in your observability platform: ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: localhost:4317 tags: environment: production cluster: kafka-01 ``` --- # Page 117: prometheus **URL**: https://docs.redpanda.com/connect/components/metrics/prometheus.md --- # prometheus --- title: prometheus latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/prometheus page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/prometheus.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/prometheus.adoc description: Host endpoints (/metrics and /stats) for Prometheus scraping. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Host endpoints (`/metrics` and `/stats`) for Prometheus scraping. ### Common ```yml metrics: prometheus: {} ``` ### Advanced ```yml metrics: prometheus: use_histogram_timing: false histogram_timing_seconds_suffix: false histogram_buckets: [] summary_quantiles_objectives: - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 add_process_metrics: false add_go_metrics: false push_url: "" # No default (optional) push_interval: "" # No default (optional) push_job_name: benthos_push push_basic_auth: username: "" password: "" file_output_path: "" ``` ## [](#fields)Fields ### [](#add_go_metrics)`add_go_metrics` Whether to export Go runtime metrics such as GC pauses in addition to Redpanda Connect metrics. **Type**: `bool` **Default**: `false` ### [](#add_process_metrics)`add_process_metrics` Whether to export process metrics such as CPU and memory usage in addition to Redpanda Connect metrics. **Type**: `bool` **Default**: `false` ### [](#file_output_path)`file_output_path` An optional file path to write all prometheus metrics on service shutdown. **Type**: `string` **Default**: `""` ### [](#histogram_buckets)`histogram_buckets[]` Timing metrics histogram buckets (in seconds). If left empty defaults to DefBuckets ([https://pkg.go.dev/github.com/prometheus/client\_golang/prometheus#pkg-variables](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#pkg-variables)). Applicable when `use_histogram_timing` is set to `true`. Requires version 3.63.0 or later. **Type**: `array` **Default**: `[]` ### [](#histogram_timing_seconds_suffix)`histogram_timing_seconds_suffix` Whether to rewrite a `_ns` timing metric name suffix to `_seconds` when `use_histogram_timing` is `true` (for example `processor_latency_ns` becomes `processor_latency_seconds`), reflecting that histogram timings are recorded in seconds. This also keeps the histogram series distinct from the summary series emitted when `use_histogram_timing` is `false`, which avoids Prometheus remote-write "multiple metric kinds" rejections when a fleet mixes the setting. Enabling this renames existing histogram timing series, so dashboards and alerts that query the `_ns` names must be updated. Has no effect when `use_histogram_timing` is `false`. Requires version 4.111.1 or later. **Type**: `bool` **Default**: `false` ### [](#push_basic_auth)`push_basic_auth` The Basic Authentication credentials. **Type**: `object` ### [](#push_basic_auth-password)`push_basic_auth.password` The Basic Authentication password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#push_basic_auth-username)`push_basic_auth.username` The Basic Authentication username. **Type**: `string` **Default**: `""` ### [](#push_interval)`push_interval` The period of time between each push when sending metrics to a Push Gateway. **Type**: `string` ### [](#push_job_name)`push_job_name` An identifier for push jobs. **Type**: `string` **Default**: `benthos_push` ### [](#push_url)`push_url` An optional [Push Gateway URL](#push-gateway) to push metrics to. **Type**: `string` ### [](#summary_quantiles_objectives)`summary_quantiles_objectives[]` A list of timing metrics summary buckets (as quantiles). Applicable when `use_histogram_timing` is set to `false`. Requires version 4.23.0 or later. **Type**: `array` **Default**: ```yaml - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 ``` ```yaml # Examples: summary_quantiles_objectives: - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 ``` ### [](#summary_quantiles_objectives-error)`summary_quantiles_objectives[].error` Permissible margin of error for quantile calculations. Precise calculations in a streaming context (without prior knowledge of the full dataset) can be resource-intensive. To balance accuracy with computational efficiency, an error margin is introduced. For instance, if the 90th quantile (`0.9`) is determined to be `100ms` with a 1% error margin (`0.01`), the true value will fall within the `[99ms, 101ms]` range.) **Type**: `float` **Default**: `0` ### [](#summary_quantiles_objectives-quantile)`summary_quantiles_objectives[].quantile` Quantile value. **Type**: `float` **Default**: `0` ### [](#use_histogram_timing)`use_histogram_timing` Whether to export timing metrics as a histogram, if `false` a summary is used instead. When exporting histogram timings the delta values are converted from nanoseconds into seconds in order to better fit within bucket definitions. For more information on histograms and summaries refer to: [https://prometheus.io/docs/practices/histograms/](https://prometheus.io/docs/practices/histograms/). Instances that report to the same destination must use the same value, unless `histogram_timing_seconds_suffix` is enabled. See [Timing metrics](#timing-metrics). Requires version 3.63.0 or later. **Type**: `bool` **Default**: `false` ## [](#push-gateway)Push gateway The field `push_url` is optional and when set will trigger a push of metrics to a [Prometheus Push Gateway](https://prometheus.io/docs/instrumenting/pushing/) once Redpanda Connect shuts down. It is also possible to specify a `push_interval` which results in periodic pushes. The Push Gateway is useful for when Redpanda Connect instances are short lived. Do not include the "/metrics/jobs/…​" path in the push URL. If the Push Gateway requires HTTP Basic Authentication it can be configured with `push_basic_auth`. --- # Page 118: statsd **URL**: https://docs.redpanda.com/connect/components/metrics/statsd.md --- # statsd --- title: statsd latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metrics/statsd page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/statsd.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/statsd.adoc description: Pushes metrics using the StatsD protocol. Supported tagging formats are 'none', 'datadog' and 'influxdb'. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Pushes metrics using the [StatsD protocol](https://github.com/statsd/statsd). Supported tagging formats are 'none', 'datadog' and 'influxdb'. #### Common ```yml metrics: statsd: address: "" # No default (required) flush_period: 100ms tag_format: none ``` #### Advanced ```yml metrics: statsd: address: "" # No default (required) flush_period: 100ms tag_format: none tags: {} ``` ## [](#fields)Fields ### [](#address)`address` The address to send metrics to. **Type**: `string` ### [](#flush_period)`flush_period` The time interval between metrics flushes. **Type**: `string` **Default**: `100ms` ### [](#tag_format)`tag_format` Metrics tagging is supported in a variety of formats. **Type**: `string` **Default**: `none` **Options**: `none`, `datadog`, `influxdb` ### [](#tags)`tags` Global tags added to each metric. Requires version 4.76.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: tags: hostname: localhost zone: danger ``` --- # Page 119: Outputs **URL**: https://docs.redpanda.com/connect/components/outputs/about.md --- # Outputs --- title: Outputs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/about.adoc description: Learn how outputs work in Redpanda Connect. An output is a sink where consumed data is sent after an optional array of processors. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- An output is a sink where we wish to send our consumed data after applying an optional array of [processors](https://docs.redpanda.com/connect/components/processors/about/). Only one output is configured at the root of a Redpanda Connect config. However, the output can be a [broker](https://docs.redpanda.com/connect/components/outputs/broker/) which combines multiple outputs under a chosen brokering pattern, or a [switch](https://docs.redpanda.com/connect/components/outputs/switch/) which is used to multiplex against different outputs. An output config section looks like this: ```yaml output: label: my_s3_output aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' # Optional list of processing steps processors: - mapping: '{"message":this,"meta":{"link_count":this.links.length()}}' ``` ## [](#back-pressure)Back pressure Redpanda Connect outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Redpanda Connect will gracefully stop consuming until the issue is resolved. ## [](#retries)Retries When a Redpanda Connect output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka). It’s possible to instead have Redpanda Connect indefinitely retry an output until success with a [`retry`](https://docs.redpanda.com/connect/components/outputs/retry/) output. Some other outputs, such as the [`broker`](https://docs.redpanda.com/connect/components/outputs/broker/), might also retry indefinitely depending on their configuration. ## [](#dead-letter-queues)Dead letter queues It’s possible to create fallback outputs for when an output target fails using a [`fallback`](https://docs.redpanda.com/connect/components/outputs/fallback/) output: ```yaml output: fallback: - aws_sqs: url: https://sqs.us-west-2.amazonaws.com/TODO/TODO max_in_flight: 20 - http_client: url: http://backup:1234/dlq verb: POST ``` ## [](#multiplexing-outputs)Multiplexing outputs There are a few different ways of multiplexing in Redpanda Connect, here’s a quick run through: ### [](#interpolation-multiplexing)Interpolation multiplexing Some output fields support [field interpolation](https://docs.redpanda.com/connect/configuration/interpolation/), which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service. For example, multiplexing against Kafka topics is a common pattern: ```yaml output: kafka: addresses: [ TODO:6379 ] topic: ${! meta("target_topic") } ``` Refer to the field documentation for a given output to see if it support interpolation. ### [](#switch-multiplexing)Switch multiplexing A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the [`switch` output](https://docs.redpanda.com/connect/components/outputs/switch/): ```yaml output: switch: cases: - check: this.type == "foo" output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - check: this.type == "bar" output: gcp_pubsub: project: dealing_with_mike topic: mikes_bars - output: redis_streams: url: tcp://localhost:6379 stream: everything_else processors: - mapping: | root = this root.type = this.type.not_null() | "unknown" ``` ## [](#labels)Labels Outputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple outputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](https://docs.redpanda.com/connect/components/metrics/about/). ## [](#categories)Categories ### Services Outputs that write to storage or message streaming services. [**RabbitMQ**](https://docs.redpanda.com/connect/components/outputs/amqp_0_9/) [**RabbitMQ**](https://docs.redpanda.com/connect/components/outputs/amqp_1/) [**Arc**](https://docs.redpanda.com/connect/components/outputs/arc/) [**AWS DynamoDB**](https://docs.redpanda.com/connect/components/outputs/aws_dynamodb/) [**AWS Kinesis**](https://docs.redpanda.com/connect/components/outputs/aws_kinesis/) [**AWS Kinesis Firehose**](https://docs.redpanda.com/connect/components/outputs/aws_kinesis_firehose/) [**AWS S3**](https://docs.redpanda.com/connect/components/outputs/aws_s3/) [**AWS SNS**](https://docs.redpanda.com/connect/components/outputs/aws_sns/) [**AWS SQS**](https://docs.redpanda.com/connect/components/outputs/aws_sqs/) [**Azure Blob Storage**](https://docs.redpanda.com/connect/components/outputs/azure_blob_storage/) [**Microsoft Azure**](https://docs.redpanda.com/connect/components/outputs/azure_data_lake_gen2/) [**Azure Queue Storage**](https://docs.redpanda.com/connect/components/outputs/azure_queue_storage/) [**Azure Table Storage**](https://docs.redpanda.com/connect/components/outputs/azure_table_storage/) [**beanstalkd**](https://docs.redpanda.com/connect/components/outputs/beanstalkd/) [**cache**](https://docs.redpanda.com/connect/components/outputs/cache/) [**cypher**](https://docs.redpanda.com/connect/components/outputs/cypher/) [**discord**](https://docs.redpanda.com/connect/components/outputs/discord/) [**doris\_stream\_load**](https://docs.redpanda.com/connect/components/outputs/doris_stream_load/) [**elasticsearch\_v8**](https://docs.redpanda.com/connect/components/outputs/elasticsearch_v8/) [**elasticsearch\_v9**](https://docs.redpanda.com/connect/components/outputs/elasticsearch_v9/) [**GCP BigQuery**](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery/) [**GCP BigQuery**](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/) [**GCP Cloud Storage**](https://docs.redpanda.com/connect/components/outputs/gcp_cloud_storage/) [**GCP PubSub**](https://docs.redpanda.com/connect/components/outputs/gcp_pubsub/) [**hdfs**](https://docs.redpanda.com/connect/components/outputs/hdfs/) [**Apache Iceberg**](https://docs.redpanda.com/connect/components/outputs/iceberg/) [**MongoDB**](https://docs.redpanda.com/connect/components/outputs/mongodb/) [**mqtt**](https://docs.redpanda.com/connect/components/outputs/mqtt/) [**NATS**](https://docs.redpanda.com/connect/components/outputs/nats/) [**NATS JetStream**](https://docs.redpanda.com/connect/components/outputs/nats_jetstream/) [**NATS KV**](https://docs.redpanda.com/connect/components/outputs/nats_kv/) [**NATS Stream**](https://docs.redpanda.com/connect/components/outputs/nats_stream/) [**nsq**](https://docs.redpanda.com/connect/components/outputs/nsq/) [**ockam\_kafka**](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/) [**opensearch**](https://docs.redpanda.com/connect/components/outputs/opensearch/) [**OpenTelemetry**](https://docs.redpanda.com/connect/components/outputs/otlp_grpc/) [**OpenTelemetry**](https://docs.redpanda.com/connect/components/outputs/otlp_http/) [**Apache Pulsar**](https://docs.redpanda.com/connect/components/outputs/pulsar/) [**pusher**](https://docs.redpanda.com/connect/components/outputs/pusher/) [**questdb**](https://docs.redpanda.com/connect/components/outputs/questdb/) [**Redis Hash**](https://docs.redpanda.com/connect/components/outputs/redis_hash/) [**Redis List**](https://docs.redpanda.com/connect/components/outputs/redis_list/) [**Redis PubSub**](https://docs.redpanda.com/connect/components/outputs/redis_pubsub/) [**Redis Streams**](https://docs.redpanda.com/connect/components/outputs/redis_streams/) [**redpanda**](https://docs.redpanda.com/connect/components/outputs/redpanda/) [**redpanda\_migrator**](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) [**Salesforce**](https://docs.redpanda.com/connect/components/outputs/salesforce_sink/) [**Snowflake**](https://docs.redpanda.com/connect/components/outputs/snowflake_put/) [**Snowflake Streaming**](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) [**Splunk**](https://docs.redpanda.com/connect/components/outputs/splunk_hec/) [**SQL**](https://docs.redpanda.com/connect/components/outputs/sql_insert/) [**SQL**](https://docs.redpanda.com/connect/components/outputs/sql_raw/) [**timeplus**](https://docs.redpanda.com/connect/components/outputs/timeplus/) ### AWS Outputs that write to Amazon Web Services products. [**AWS DynamoDB**](https://docs.redpanda.com/connect/components/outputs/aws_dynamodb/) [**AWS Kinesis**](https://docs.redpanda.com/connect/components/outputs/aws_kinesis/) [**AWS Kinesis Firehose**](https://docs.redpanda.com/connect/components/outputs/aws_kinesis_firehose/) [**AWS S3**](https://docs.redpanda.com/connect/components/outputs/aws_s3/) [**AWS SNS**](https://docs.redpanda.com/connect/components/outputs/aws_sns/) [**AWS SQS**](https://docs.redpanda.com/connect/components/outputs/aws_sqs/) ### Azure Outputs that write to Microsoft Azure services. [**Azure Blob Storage**](https://docs.redpanda.com/connect/components/outputs/azure_blob_storage/) [**Microsoft Azure**](https://docs.redpanda.com/connect/components/outputs/azure_cosmosdb/) [**Microsoft Azure**](https://docs.redpanda.com/connect/components/outputs/azure_data_lake_gen2/) [**Azure Queue Storage**](https://docs.redpanda.com/connect/components/outputs/azure_queue_storage/) [**Azure Table Storage**](https://docs.redpanda.com/connect/components/outputs/azure_table_storage/) ### Utility Outputs that provide utility by combining/wrapping other outputs. [**broker**](https://docs.redpanda.com/connect/components/outputs/broker/) [**drop**](https://docs.redpanda.com/connect/components/outputs/drop/) [**drop\_on**](https://docs.redpanda.com/connect/components/outputs/drop_on/) [**dynamic**](https://docs.redpanda.com/connect/components/outputs/dynamic/) [**fallback**](https://docs.redpanda.com/connect/components/outputs/fallback/) [**inproc**](https://docs.redpanda.com/connect/components/outputs/inproc/) [**reject**](https://docs.redpanda.com/connect/components/outputs/reject/) [**reject\_errored**](https://docs.redpanda.com/connect/components/outputs/reject_errored/) [**resource**](https://docs.redpanda.com/connect/components/outputs/resource/) [**retry**](https://docs.redpanda.com/connect/components/outputs/retry/) [**subprocess**](https://docs.redpanda.com/connect/components/outputs/subprocess/) [**switch**](https://docs.redpanda.com/connect/components/outputs/switch/) [**sync\_response**](https://docs.redpanda.com/connect/components/outputs/sync_response/) ### Integration [**couchbase**](https://docs.redpanda.com/connect/components/outputs/couchbase/) [**schema\_registry**](https://docs.redpanda.com/connect/components/outputs/schema_registry/) ### AI [**cyborgdb**](https://docs.redpanda.com/connect/components/outputs/cyborgdb/) [**pinecone**](https://docs.redpanda.com/connect/components/outputs/pinecone/) [**qdrant**](https://docs.redpanda.com/connect/components/outputs/qdrant/) ### Social Outputs that write to social applications and services. [**discord**](https://docs.redpanda.com/connect/components/outputs/discord/) ### Local Outputs that write to the local machine/filesystem. [**file**](https://docs.redpanda.com/connect/components/outputs/file/) [**stdout**](https://docs.redpanda.com/connect/components/outputs/stdout/) ### GCP Outputs that write to Google Cloud Platform services. [**GCP BigQuery**](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery/) [**GCP BigQuery**](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/) [**GCP Cloud Storage**](https://docs.redpanda.com/connect/components/outputs/gcp_cloud_storage/) [**GCP PubSub**](https://docs.redpanda.com/connect/components/outputs/gcp_pubsub/) ### Network Outputs that write directly to low level network protocols. [**HTTP**](https://docs.redpanda.com/connect/components/outputs/http_client/) [**HTTP**](https://docs.redpanda.com/connect/components/outputs/http_server/) [**nanomsg**](https://docs.redpanda.com/connect/components/outputs/nanomsg/) [**sftp**](https://docs.redpanda.com/connect/components/outputs/sftp/) [**socket**](https://docs.redpanda.com/connect/components/outputs/socket/) [**websocket**](https://docs.redpanda.com/connect/components/outputs/websocket/) [**zmq4**](https://docs.redpanda.com/connect/components/outputs/zmq4/) ### [**Slack Reaction**](https://docs.redpanda.com/connect/components/outputs/slack_reaction/) --- # Page 120: amqp_0_9 **URL**: https://docs.redpanda.com/connect/components/outputs/amqp_0_9.md --- # amqp_0_9 --- title: amqp_0_9 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/amqp_0_9 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/amqp_0_9.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/amqp_0_9.adoc description: Sends messages to an AMQP (0.91) exchange. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to an AMQP (0.91) exchange. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. #### Common ```yml output: label: "" amqp_0_9: urls: [] # No default (required) exchange: "" # No default (required) key: "" type: "" metadata: exclude_prefixes: [] max_in_flight: 64 ``` #### Advanced ```yml output: label: "" amqp_0_9: urls: [] # No default (required) exchange: "" # No default (required) exchange_declare: enabled: false type: direct durable: true arguments: {} # No default (optional) key: "" type: "" content_type: application/octet-stream content_encoding: "" correlation_id: "" reply_to: "" expiration: "" message_id: "" user_id: "" app_id: "" metadata: exclude_prefixes: [] priority: "" max_in_flight: 64 persistent: false mandatory: false immediate: false timeout: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` The metadata fields from each message are delivered as headers. TLS is automatically enabled when connecting to an `amqps` URL. However, you can customize [TLS settings](#tls) if required. You can use [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to dynamically set values for the following fields: `key`, `exchange`, and `type`. ## [](#fields)Fields ### [](#app_id)`app_id` Set an application ID for each message using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.22.0 or later. **Type**: `string` **Default**: `""` ### [](#content_encoding)`content_encoding` The content encoding attribute of each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The MIME type of each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#correlation_id)`correlation_id` Set a unique correlation ID for each message using a dynamic interpolated expression to help match messages to responses. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.22.0 or later. **Type**: `string` **Default**: `""` ### [](#exchange)`exchange` The AMQP exchange to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#exchange_declare)`exchange_declare` Declares the target exchange (`exchange`) to check whether an exchange with the specified name exists and is configured correctly. If the exchange exists, the declaration verifies that the fields specified in this object match its properties. If the target exchange does not exist, this output creates it. **Type**: `object` ### [](#exchange_declare-arguments)`exchange_declare.arguments` Arguments for server-specific implementations of the exchange (optional). You can use arguments to configure additional parameters for exchange types that require them. Requires version 4.46.0 or later. **Type**: `object` ```yaml # Examples: arguments: alternate-exchange: my-ae ``` ### [](#exchange_declare-durable)`exchange_declare.durable` Whether the declared exchange is durable. **Type**: `bool` **Default**: `true` ### [](#exchange_declare-enabled)`exchange_declare.enabled` Whether to enable exchange declaration. **Type**: `bool` **Default**: `false` ### [](#exchange_declare-type)`exchange_declare.type` The type of the exchange, which determines how messages are routed to queues. For `topic` exchanges, routing keys are matched as lists of words separated by dots (`.`). **Type**: `string` **Default**: `direct` **Options**: `direct`, `fanout`, `topic`, `headers`, `x-custom` ### [](#expiration)`expiration` Set the TTL of each message in milliseconds. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.22.0 or later. **Type**: `string` **Default**: `""` ### [](#immediate)`immediate` Whether to set the immediate flag on published messages. When set to `true`, if there are no active consumers for a queue, the message is dropped instead of waiting. **Type**: `bool` **Default**: `false` ### [](#key)`key` The binding key to set for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#mandatory)`mandatory` Whether to set the mandatory flag on published messages. When set to `true`, a published message that cannot be routed to any queues is returned to the sender. **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_id)`message_id` Set a message ID for each message using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.22.0 or later. **Type**: `string` **Default**: `""` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#persistent)`persistent` Whether to store delivered messages on disk. By default, message delivery is transient. **Type**: `bool` **Default**: `false` ### [](#priority)`priority` Set the priority of each message using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: priority: "0" # --- priority: ${! meta("amqp_priority") } # --- priority: ${! json("doc.priority") } ``` ### [](#reply_to)`reply_to` Set the name of the queue to which responses are sent using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.22.0 or later. **Type**: `string` **Default**: `""` ### [](#timeout)`timeout` The maximum period to wait for a message acknowledgment before abandoning it and attempting a resend. If this value is not set, the system waits indefinitely. Requires version 4.5.0 or later. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#type)`type` A custom message type to set for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#urls)`urls[]` A list of AMQP 0.9.1 broker URLs to connect to, in the form `amqp://user:password@host:port/vhost`. Each URL is tried in order until a connection is established, and that URL is then used until the connection is closed. When `tls` is enabled and a URL has no user credentials, the connection authenticates with the SASL `EXTERNAL` mechanism using the client certificate. If an item in the list contains commas, it is split into multiple URLs. Requires version 3.58.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` ### [](#user_id)`user_id` Set the user ID to the name of the publisher. If this property is set by a publisher, its value must match the name of the user that opened the connection. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.22.0 or later. **Type**: `string` **Default**: `""` --- # Page 121: amqp_1 **URL**: https://docs.redpanda.com/connect/components/outputs/amqp_1.md --- # amqp_1 --- title: amqp_1 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/amqp_1 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/amqp_1.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/amqp_1.adoc description: Sends messages to an AMQP (1.0) server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to an AMQP (1.0) server. #### Common ```yml output: label: "" amqp_1: urls: [] # No default (optional) target_address: "" max_in_flight: 64 metadata: exclude_prefixes: [] ``` #### Advanced ```yml output: label: "" amqp_1: urls: [] # No default (optional) target_address: "" max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] application_properties_map: "" # No default (optional) sasl: mechanism: none user: "" password: "" metadata: exclude_prefixes: [] content_type: opaque_binary persistent: false target_capabilities: [] # No default (optional) message_properties_to: "" # No default (optional) message_properties_message_id: "" # No default (optional) message_properties_correlation_id: "" # No default (optional) message_properties_subject: "" # No default (optional) message_properties_reply_to: "" # No default (optional) message_properties_group_id: "" # No default (optional) message_properties_group_sequence: "" # No default (optional) message_properties_reply_to_group_id: "" # No default (optional) message_properties_user_id: "" # No default (optional) message_properties_content_type: "" # No default (optional) message_properties_content_encoding: "" # No default (optional) ``` ## [](#metadata)Metadata Message metadata is added to each AMQP message as string annotations. In order to control which metadata keys are added use the `metadata` config field. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#application_properties_map)`application_properties_map` An optional Bloblang mapping that can be defined in order to set the `application-properties` on output messages. Requires version 4.11.0 or later. **Type**: `string` ### [](#content_type)`content_type` The content type of the message body. Set this field value to `string` to transfer each message as an AMQP string. Consider using the `string` option if you want to write UTF-8 string messages, such as JSON messages, to your data destination. Requires version 4.45.0 or later. **Type**: `string` **Default**: `opaque_binary` **Options**: `opaque_binary`, `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_properties_content_encoding)`message_properties_content_encoding` Set the content-encoding property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#message_properties_content_type)`message_properties_content_type` Set the content-type property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ```yaml # Examples: message_properties_content_type: application/json # --- message_properties_content_type: text/plain; charset=utf-8 ``` ### [](#message_properties_correlation_id)`message_properties_correlation_id` Set the correlation-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. Purely numeric values are sent as uint64 on the wire. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ```yaml # Examples: message_properties_correlation_id: ${! meta("amqp_correlation_id") } ``` ### [](#message_properties_group_id)`message_properties_group_id` Set the group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#message_properties_group_sequence)`message_properties_group_sequence` Set the group-sequence property on outgoing AMQP messages. Must be a valid uint32 value. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#message_properties_message_id)`message_properties_message_id` Set the message-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. Purely numeric values are sent as uint64 on the wire. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ```yaml # Examples: message_properties_message_id: ${! uuid_v4() } # --- message_properties_message_id: ${! meta("amqp_message_id") } ``` ### [](#message_properties_reply_to)`message_properties_reply_to` Set the reply-to property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#message_properties_reply_to_group_id)`message_properties_reply_to_group_id` Set the reply-to-group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#message_properties_subject)`message_properties_subject` Set the subject property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#message_properties_to)`message_properties_to` The field specifies the node that is the intended destination of the message, which may differ from the node currently receiving the transfer. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.70.0 or later. **Type**: `string` ```yaml # Examples: message_properties_to: amqp://localhost:5672/ # --- message_properties_to: ${! meta("target_address") } ``` ### [](#message_properties_user_id)`message_properties_user_id` Set the user-id property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.87.0 or later. **Type**: `string` ### [](#metadata-2)`metadata` Specify which message metadata keys are added to each AMQP 1.0 message as message annotations. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#persistent)`persistent` If set to true, the message will be marked as persistent, ensuring it is stored durably and not lost if an intermediary (such as a broker) restarts. By default, messages are not durable. Requires version 4.70.0 or later. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism to use. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | anonymous | Anonymous SASL authentication. | | none | No SASL based authentication. | | plain | Plain text SASL authentication. | ### [](#sasl-password)`sasl.password` A SASL plain text password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-user)`sasl.user` A SASL plain text username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#target_address)`target_address` The target address to write to. When left empty, the output uses the Anonymous Terminus pattern where the destination is specified per-message using `message_properties_to`. **Type**: `string` **Default**: `""` ```yaml # Examples: target_address: /foo # --- target_address: queue:/bar # --- target_address: topic:/baz # --- target_address: "" ``` ### [](#target_capabilities)`target_capabilities[]` Lists the extension capabilities the sender desires from the target, such as support for queues, topics, durability, sharing, or temporary destinations. Requires version 4.70.0 or later. **Type**: `array` ```yaml # Examples: target_capabilities: - queue # --- target_capabilities: - topic # --- target_capabilities: - queue - topic ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of AMQP 1.0 broker URLs to connect to. Each URL is tried in order until a connection is established, and that URL is then used until the connection is closed. This field replaces the deprecated `url` field, which is only read when `urls` is empty. If an item in the list contains commas, it is split into multiple URLs. Requires version 4.23.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 122: arc **URL**: https://docs.redpanda.com/connect/components/outputs/arc.md --- # arc --- title: arc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/arc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/arc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/arc.adoc description: Writes data to an Arc database via the msgpack ingestion endpoint. page-git-created-date: "2026-04-20" page-git-modified-date: "2026-10-10" --- Writes data to an Arc database via the msgpack ingestion endpoint. Introduced in version 4.88.0. This output sends data to an [Arc](https://github.com/Basekick-Labs/arc) columnar analytical database using its high-performance MessagePack ingestion endpoint. Arc supports two payload formats: - **columnar** (default): Transposes batched messages into column arrays. This is the recommended format, offering significantly faster ingestion. - **row**: Sends each message as an individual row record with fields and optional tags. Data is encoded as MessagePack and optionally compressed with zstd (recommended) or gzip before being sent to the Arc endpoint. > 📝 **NOTE** > > In columnar mode, all messages within a single batch must have the same set of fields. Arc validates that all column arrays have equal length and rejects batches with mismatched columns. Schema evolution across separate batches is fully supported. Use row format if messages within a batch have varying schemas. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" arc: base_url: "" # No default (required) timeout: 5s token: "" # No default (optional) database: default measurement: "" # No default (required) format: columnar tags_mapping: "" # No default (optional) compression: zstd max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" arc: base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 token: "" # No default (optional) database: default measurement: "" # No default (required) format: columnar timestamp_field: "" timestamp_unit: auto tags_mapping: "" # No default (optional) compression: zstd max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` The base URL of the target service (for example, `[https://api.example.com](https://api.example.com)`). TLS is enabled automatically for `https` URLs. **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#compression)`compression` Compression algorithm for the request body. `zstd` is recommended for best decompression performance in Arc. **Type**: `string` **Default**: `zstd` **Options**: `zstd`, `gzip`, `none` ### [](#database)`database` The target database name. **Type**: `string` **Default**: `default` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#format)`format` The payload format. `columnar` transposes batch messages into column arrays for best performance. `row` sends each message as an individual record. **Type**: `string` **Default**: `columnar` **Options**: `columnar`, `row` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#measurement)`measurement` The measurement (table) name. Supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: measurement: cpu_metrics # --- measurement: ${!metadata("measurement")} # --- measurement: ${!json("type")} ``` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#tags_mapping)`tags_mapping` An optional Bloblang mapping to extract tags from each message. Only used in `row` format. The result must be a `map[string]string`. **Type**: `string` ```yaml # Examples: tags_mapping: 'root = {"host": this.hostname, "region": this.region}' ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#timestamp_field)`timestamp_field` The field name within each message containing the timestamp. If empty, the current time is used. Supports Unix timestamps and RFC3339 strings. **Type**: `string` **Default**: `""` ### [](#timestamp_unit)`timestamp_unit` The unit of a numeric timestamp field. `auto` detects the unit based on magnitude. Ignored when `timestamp_field` is empty. **Type**: `string` **Default**: `auto` **Options**: `us`, `ms`, `s`, `ns`, `auto` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#token)`token` Bearer token for authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` --- # Page 123: aws_dynamodb **URL**: https://docs.redpanda.com/connect/components/outputs/aws_dynamodb.md --- # aws_dynamodb --- title: aws_dynamodb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/aws_dynamodb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_dynamodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_dynamodb.adoc description: Inserts items into a DynamoDB table. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Inserts items into a DynamoDB table. Introduced in version 3.36.0. The field `string_columns` is a map of column names to string values, where the values are [function interpolated](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) per message of a batch. This allows you to populate string columns of an item by extracting fields within the document payload or metadata like follows: ```yml string_columns: id: ${!json("id")} title: ${!json("body.title")} topic: ${!meta("kafka_topic")} full_content: ${!content()} ``` The field `json_map_columns` is a map of column names to json paths, where the [dot path](https://docs.redpanda.com/connect/configuration/field_paths/) is extracted from each document and converted into a map value. Both an empty path and the path `.` are interpreted as the root of the document. This allows you to populate map columns of an item like follows: ```yml json_map_columns: user: path.to.user whole_document: . ``` A column name can be empty: ```yml json_map_columns: "": . ``` In which case the top level document fields will be written at the root of the item, potentially overwriting previously defined column values. If a path is not found within a document the column will not be populated. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" aws_dynamodb: table: "" # No default (required) string_columns: {} json_map_columns: {} max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" aws_dynamodb: table: "" # No default (required) string_columns: {} json_map_columns: {} ttl: "" ttl_key: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 3 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#backoff)`backoff` The exponential backoff between retry attempts. The wait starts at `initial_interval` and grows with each failed attempt, up to `max_interval`. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#json_map_columns)`json_map_columns` A map of column keys to [field paths](https://docs.redpanda.com/connect/configuration/field_paths/) pointing to value data within messages. **Type**: `object` **Default**: `{}` ```yaml # Examples: json_map_columns: user: path.to.user whole_document: . # --- json_map_columns: "": . ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `3` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#string_columns)`string_columns` A map of column keys to string values to store. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: string_columns: full_content: ${!content()} id: ${!json("id")} title: ${!json("body.title")} topic: ${!meta("kafka_topic")} ``` ### [](#table)`table` The table to store messages in. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#ttl)`ttl` An optional TTL to set for items, calculated from the moment the message is sent. **Type**: `string` **Default**: `""` ### [](#ttl_key)`ttl_key` The column key to place the TTL value within. The value is the expiry time as a Unix timestamp in seconds, stored as a number attribute, which is the format that DynamoDB TTL requires. A TTL value is only written when both this field and a TTL duration are set. **Type**: `string` **Default**: `""` --- # Page 124: aws_kinesis_firehose **URL**: https://docs.redpanda.com/connect/components/outputs/aws_kinesis_firehose.md --- # aws_kinesis_firehose --- title: aws_kinesis_firehose latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/aws_kinesis_firehose page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_kinesis_firehose.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_kinesis_firehose.adoc description: Sends messages to a Kinesis Firehose delivery stream. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to a Kinesis Firehose delivery stream. Introduced in version 3.36.0. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" aws_kinesis_firehose: stream: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" aws_kinesis_firehose: stream: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#backoff)`backoff` The exponential backoff between retry attempts. The wait starts at `initial_interval` and grows with each failed attempt, up to `max_interval`. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#stream)`stream` The stream to publish messages to. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 125: aws_kinesis **URL**: https://docs.redpanda.com/connect/components/outputs/aws_kinesis.md --- # aws_kinesis --- title: aws_kinesis latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/aws_kinesis page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_kinesis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_kinesis.adoc description: Sends messages to a Kinesis stream. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to a Kinesis stream. Introduced in version 3.36.0. Both the `partition_key`(required) and `hash_key` (optional) fields can be dynamically set using function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When sending batched messages the interpolations are performed per message part. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" aws_kinesis: stream: "" # No default (required) partition_key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" aws_kinesis: stream: "" # No default (required) partition_key: "" # No default (required) hash_key: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#backoff)`backoff` The exponential backoff between retry attempts. The wait starts at `initial_interval` and grows with each failed attempt, up to `max_interval`. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#hash_key)`hash_key` A optional hash key for partitioning messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#partition_key)`partition_key` A required key for partitioning messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#stream)`stream` The stream to publish messages to. Streams can either be specified by their name or full ARN. **Type**: `string` ```yaml # Examples: stream: foo # --- stream: arn:aws:kinesis:*:111122223333:stream/my-stream ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 126: aws_s3 **URL**: https://docs.redpanda.com/connect/components/outputs/aws_s3.md --- # aws_s3 --- title: aws_s3 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/aws_s3 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_s3.adoc description: Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded with the path specified with the path field. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Uploads messages to an Amazon S3 bucket as objects, using the path specified in the `path` field. Introduced in version 3.36.0. #### Common ```yml output: label: "" aws_s3: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt tags: {} content_type: application/octet-stream metadata: exclude_prefixes: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" aws_s3: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt tags: {} content_type: application/octet-stream content_encoding: "" cache_control: "" content_disposition: "" content_language: "" website_redirect_location: "" metadata: exclude_prefixes: [] storage_class: STANDARD kms_key_id: "" checksum_algorithm: "" server_side_encryption: "" force_path_style_urls: false max_in_flight: 64 timeout: 5s object_canned_acl: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` To use a different path for each object, use [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which is evaluated for each message in a batch. ## [](#metadata)Metadata Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](https://docs.redpanda.com/connect/configuration/metadata/). ## [](#tags)Tags The `tags` field accepts key/value pairs to attach to objects as tags, and the values support [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries): ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz tags: Key1: Value1 Timestamp: ${!meta("Timestamp")} ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#batching)Batching It’s common to want to upload messages to S3 as batched archives. The easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](https://docs.redpanda.com/connect/components/processors/archive/) or [`compress`](https://docs.redpanda.com/connect/components/processors/compress/) processor. For example, the following configuration uploads messages as a `.tar.gz` archive of documents: ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz batching: count: 100 period: 10s processors: - archive: format: tar - compress: algorithm: gzip ``` This configuration uploads JSON documents as a single large document containing an array of objects: ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.json batching: count: 100 processors: - archive: format: json_array ``` ## [](#bucket-name-format)Bucket name format The `bucket` field accepts a bucket name only, not an ARN. For example, use `my-bucket`, not `arn:aws:s3:::my-bucket`. ## [](#s3-compatible-storage)S3-compatible storage The `endpoint` and `force_path_style_urls` fields let you connect to S3-compatible storage services such as Cloudflare R2, MinIO, or DigitalOcean Spaces. For Cloudflare R2, set `endpoint` to your account endpoint URL and enable `force_path_style_urls`: ```yaml output: aws_s3: bucket: r2-bucket path: ${!uuid_v4()}.json endpoint: https://.r2.cloudflarestorage.com force_path_style_urls: true region: auto credentials: id: secret: ``` Find your account ID in the Cloudflare dashboard under **R2 > Overview > Account Details**. Generate API credentials under **R2 > Manage R2 API Tokens**. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The name of the bucket to upload messages to, such as `my-bucket`. Use the bucket name, not the bucket ARN. **Type**: `string` ### [](#cache_control)`cache_control` The cache control to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#checksum_algorithm)`checksum_algorithm` The algorithm used to create the checksum for each object, which Amazon S3 uses to validate the object during upload. Requires version 4.38.0 or later. **Type**: `string` **Default**: `""` **Options**: `CRC32`, `CRC32C`, `SHA1`, `SHA256` ### [](#content_disposition)`content_disposition` The content disposition to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_encoding)`content_encoding` An optional `Content-Encoding` value, such as `gzip`, to store with each uploaded S3 object. When empty, S3 stores the object without a content encoding. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_language)`content_language` The content language to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The MIME type to store as the `Content-Type` of each uploaded S3 object. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs, which helps when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#kms_key_id)`kms_key_id` An optional server-side encryption key. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata-2)`metadata` Specify which message metadata keys are stored as user-defined metadata on each S3 object, which S3 returns as `x-amz-meta-*` headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#object_canned_acl)`object_canned_acl` The object canned ACL value. Leave empty to omit the ACL from upload requests, which is required for buckets that have ACLs disabled (the AWS default since 2023). Requires version 4.56.0 or later. **Type**: `string` **Default**: `""` **Options**: `""`, `private`, `public-read`, `public-read-write`, `authenticated-read`, `aws-exec-read`, `bucket-owner-read`, `bucket-owner-full-control` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#server_side_encryption)`server_side_encryption` An optional server-side encryption algorithm. Requires version 3.63.0 or later. **Type**: `string` **Default**: `""` ### [](#storage_class)`storage_class` The storage class to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `STANDARD` **Options**: `STANDARD`, `REDUCED_REDUNDANCY`, `GLACIER`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `DEEP_ARCHIVE` ### [](#tags-2)`tags` Key/value pairs to store with the object as tags. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: tags: Key1: Value1 Timestamp: ${!meta("Timestamp")} ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period to wait for every object in a message batch to upload to S3. When the timeout is reached, the batch upload is abandoned and reattempted. **Type**: `string` **Default**: `5s` ### [](#website_redirect_location)`website_redirect_location` The website redirect location to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 127: aws_sns **URL**: https://docs.redpanda.com/connect/components/outputs/aws_sns.md --- # aws_sns --- title: aws_sns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/aws_sns page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_sns.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_sns.adoc description: Sends messages to an AWS SNS topic. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to an AWS SNS topic. Introduced in version 3.36.0. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" aws_sns: topic_arn: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) subject: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] ``` ### Advanced ```yml output: label: "" aws_sns: topic_arn: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) subject: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] timeout: 5s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#fields)Fields ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_deduplication_id)`message_deduplication_id` An optional deduplication ID for messages published to an SNS FIFO topic. SNS discards a message whose deduplication ID it has already accepted within the deduplication interval. Not required when the topic has content-based deduplication enabled. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.60.0 or later. **Type**: `string` ### [](#message_group_id)`message_group_id` An optional message group ID. Required when publishing to an SNS FIFO topic, which delivers messages that share a group ID in order. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.60.0 or later. **Type**: `string` ### [](#metadata)`metadata` Specify which metadata values are sent as SNS message attributes of type `String`. Metadata keys that are not valid SNS attribute names are skipped. Requires version 3.60.0 or later. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#subject)`subject` An optional subject to set for messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.81.0 or later. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period to wait on a publish request before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#topic_arn)`topic_arn` The topic to publish to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 128: aws_sqs **URL**: https://docs.redpanda.com/connect/components/outputs/aws_sqs.md --- # aws_sqs --- title: aws_sqs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/aws_sqs page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_sqs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_sqs.adoc description: Sends messages to an SQS queue. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to an SQS queue. Introduced in version 3.36.0. Metadata values are sent along with the payload as attributes with the data type String. If the number of metadata values in a message exceeds the message attribute limit (10) then the top ten keys ordered alphabetically will be selected. The fields `message_group_id`, `message_deduplication_id` and `delay_seconds` can be set dynamically using [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which are resolved individually for each message of a batch. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" aws_sqs: url: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) delay_seconds: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" aws_sqs: url: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) delay_seconds: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_records_per_request: 10 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#backoff)`backoff` The exponential backoff between retry attempts. The wait starts at `initial_interval` and grows with each failed attempt, up to `max_interval`. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#delay_seconds)`delay_seconds` An optional delay time in seconds for messages. The value must be between `0` and `900`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.24.0 or later. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#max_records_per_request)`max_records_per_request` The maximum number of records delivered in a single SQS request. Enter a value from `1` to `10`. Requires version 4.41.0 or later. **Type**: `int` **Default**: `10` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#message_deduplication_id)`message_deduplication_id` An optional deduplication ID for messages sent to an SQS FIFO queue. SQS accepts only the first message with a given deduplication ID within the five-minute deduplication interval. Not required when the queue has content-based deduplication enabled. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_group_id)`message_group_id` An optional message group ID. Required when sending to an SQS FIFO queue, where messages that share a group ID are received in the order they were sent. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#metadata)`metadata` Specify which metadata values are sent as SQS message attributes of type `String`. SQS allows at most 10 attributes per message, so only the first 10 matching keys in sorted order are sent. Keys that are not valid SQS attribute names are skipped. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#url)`url` The URL of the target SQS queue. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 129: azure_blob_storage **URL**: https://docs.redpanda.com/connect/components/outputs/azure_blob_storage.md --- # azure_blob_storage --- title: azure_blob_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/azure_blob_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_blob_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_blob_storage.adoc description: Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the container field. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` field. Introduced in version 3.36.0. In order to have a different path for each object you should use function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. Supports multiple authentication methods but only one of the following is required: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access via [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If multiple are set then the `storage_connection_string` is given priority. If the `storage_connection_string` does not contain the `AccountName` parameter, please specify it in the `storage_account` field. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 ``` ### Advanced ```yml output: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt blob_type: BLOCK public_access_level: PRIVATE max_in_flight: 64 ``` ## [](#fields)Fields ### [](#blob_type)`blob_type` Block and Append blobs are made up of blocks, and each blob can support up to 50,000 blocks. The default value is `BLOCK`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `BLOCK` **Options**: `BLOCK`, `APPEND` ### [](#container)`container` The container for uploading the messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: container: messages-${!timestamp("2006")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.json # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#public_access_level)`public_access_level` The container’s public access level. The default value is `PRIVATE`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `PRIVATE` **Options**: `PRIVATE`, `BLOB`, `CONTAINER` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` --- # Page 130: azure_cosmosdb **URL**: https://docs.redpanda.com/connect/components/outputs/azure_cosmosdb.md --- # azure_cosmosdb --- title: azure_cosmosdb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/azure_cosmosdb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_cosmosdb.adoc description: Creates or updates messages as JSON documents in Azure CosmosDB. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction). Introduced in version 4.25.0. When creating documents, each message must have the `id` property (case-sensitive) set (or use `auto_id: true`). It is the unique name that identifies the document, that is, no two documents share the same `id` within a logical partition. The `id` field must not exceed 255 characters. [See details](https://learn.microsoft.com/en-us/rest/api/cosmos-db/documents). The `partition_keys` field must resolve to the same value(s) across the entire message batch. ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#batching)Batching CosmosDB limits the maximum batch size to 100 messages and the payload must not exceed 2MB ([details here](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-request-limits)). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create item_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" max_in_flight: 64 ``` ### Advanced ```yml output: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create patch_operations: [] # No default (optional) patch_condition: "" # No default (optional) auto_id: true item_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` ## [](#examples)Examples ### [](#create-documents)Create documents Create new documents in the `blobfish` container with partition key `/habitat`. ```yaml output: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = json("habitat") operation: Create ``` ### [](#patch-documents)Patch documents Execute the Patch operation on documents from the `blobfish` container. ```yaml output: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: testdb container: blobfish partition_keys_map: root = json("habitat") item_id: ${! json("id") } operation: Patch patch_operations: # Add a new /diet field - operation: Add path: /diet value_map: root = json("diet") # Remove the first location from the /locations array field - operation: Remove path: /locations/0 # Add new location at the end of the /locations array field - operation: Add path: /locations/- value_map: root = "Challenger Deep" ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#auto_id)`auto_id` Automatically set the item `id` field to a random UUID v4. If the `id` field is already set, then it will not be overwritten. Setting this to `false` can improve performance, since the messages will not have to be parsed. **Type**: `bool` **Default**: `true` ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#item_id)`item_id` ID of item to replace or delete. Only used by the Replace and Delete operations This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: item_id: ${! json("id") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` Operation. **Type**: `string` **Default**: `Create` | Option | Summary | | --- | --- | | Create | Create operation. | | Delete | Delete operation. | | Patch | Patch operation. | | Replace | Replace operation. | | Upsert | Upsert operation. | ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = json("blobfish").depth ``` ### [](#patch_condition)`patch_condition` Patch operation condition. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: patch_condition: from c where not is_defined(c.blobfish) ``` ### [](#patch_operations)`patch_operations[]` Patch operations to be performed when `operation: Patch` . **Type**: `array` ### [](#patch_operations-operation)`patch_operations[].operation` Operation. **Type**: `string` **Default**: `Add` | Option | Summary | | --- | --- | | Add | Add patch operation. | | Increment | Increment patch operation. | | Remove | Remove patch operation. | | Replace | Replace patch operation. | | Set | Set patch operation. | ### [](#patch_operations-path)`patch_operations[].path` Path. **Type**: `string` ```yaml # Examples: path: /foo/bar/baz ``` ### [](#patch_operations-value_map)`patch_operations[].value_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to a value of any type that is supported by CosmosDB. **Type**: `string` ```yaml # Examples: value_map: root = "blobfish" # --- value_map: root = 41 # --- value_map: root = true # --- value_map: root = json("blobfish").depth # --- value_map: root = [1, 2, 3] ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 131: azure_data_lake_gen2 **URL**: https://docs.redpanda.com/connect/components/outputs/azure_data_lake_gen2.md --- # azure_data_lake_gen2 --- title: azure_data_lake_gen2 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/azure_data_lake_gen2 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_data_lake_gen2.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_data_lake_gen2.adoc description: Sends message parts as files to an Azure Data Lake Gen2 filesystem. Each file is uploaded with the filename specified with the path field. page-git-created-date: "2024-11-05" page-git-modified-date: "2026-10-10" --- Sends message parts as files to an [Azure Data Lake Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) file system. Each file is uploaded with the file name specified in the `path` field. Introduced in version 4.38.0. ```yml output: label: "" azure_data_lake_gen2: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" filesystem: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 ``` To specify a different [`path` value](#path) (file name) for each file, use [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Function interpolations are calculated for each message in a batch. ## [](#authentication-methods)Authentication methods This output supports multiple authentication methods. You must configure at least one method from the following list: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access using [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If you configure multiple authentication methods, the `storage_connection_string` takes precedence. ## [](#performance)Performance Sends multiple messages in flight in parallel for improved performance. You can tune the number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#filesystem)`filesystem` The name of the data lake storage file system you want to upload messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: filesystem: messages-${!timestamp("2006")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path (file name) of each message to upload to the data lake storage file system. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.json # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` --- # Page 132: azure_queue_storage **URL**: https://docs.redpanda.com/connect/components/outputs/azure_queue_storage.md --- # azure_queue_storage --- title: azure_queue_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/azure_queue_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_queue_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_queue_storage.adoc description: Sends messages to an Azure Storage Queue. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to an Azure Storage Queue. Introduced in version 3.36.0. Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. In order to set the `queue_name` you can use function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" queue_name: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" queue_name: "" # No default (required) ttl: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#queue_name)`queue_name` The name of the target Queue Storage queue. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` ### [](#ttl)`ttl` The TTL of each individual message as a duration string. Defaults to 0, meaning no retention period is set This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` --- # Page 133: azure_table_storage **URL**: https://docs.redpanda.com/connect/components/outputs/azure_table_storage.md --- # azure_table_storage --- title: azure_table_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/azure_table_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_table_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_table_storage.adoc description: Stores messages in an Azure Table Storage table. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores messages in an Azure Table Storage table. Introduced in version 3.36.0. Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. In order to set the `table_name`, `partition_key` and `row_key` you can use function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. If the `properties` are not set in the config, all the `json` fields are marshalled and stored in the table, which will be created if it does not exist. The `object` and `array` fields are marshaled as strings. For example: The JSON message: ```json { "foo": 55, "bar": { "baz": "a", "bez": "b" }, "diz": ["a", "b"] } ``` Will store in the table the following properties: ```yml foo: '55' bar: '{ "baz": "a", "bez": "b" }' diz: '["a", "b"]' ``` It’s also possible to use function interpolations to get or transform the properties values. For example: ```yml properties: device: '${! json("device") }' timestamp: '${! json("timestamp") }' ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) partition_key: "" row_key: "" properties: {} max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) partition_key: "" row_key: "" properties: {} transaction_type: INSERT max_in_flight: 64 timeout: 5s batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#partition_key)`partition_key` The partition key. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: partition_key: ${! json("date") } ``` ### [](#properties)`properties` A map of properties to store into the table. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ### [](#row_key)`row_key` The row key. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: row_key: ${! json("device")}-${!uuid_v4() } ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated, unless the connection string does not contain the `AccountName` parameter. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. This field is required if `storage_account` is not set. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` ### [](#table_name)`table_name` The table to store messages into. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table_name: ${! meta("kafka_topic") } # --- table_name: ${! json("table") } ``` ### [](#timeout)`timeout` This field has no effect. The output parses the value but does not apply it to write requests. **Type**: `string` **Default**: `5s` ### [](#transaction_type)`transaction_type` Type of transaction operation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.10.0 or later. **Type**: `string` **Default**: `INSERT` **Options**: `INSERT`, `INSERT_MERGE`, `INSERT_REPLACE`, `UPDATE_MERGE`, `UPDATE_REPLACE`, `DELETE` ```yaml # Examples: transaction_type: ${! json("operation") } # --- transaction_type: ${! meta("operation") } # --- transaction_type: INSERT ``` --- # Page 134: beanstalkd **URL**: https://docs.redpanda.com/connect/components/outputs/beanstalkd.md --- # beanstalkd --- title: beanstalkd latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/beanstalkd page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/beanstalkd.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/beanstalkd.adoc description: Write messages to a Beanstalkd queue. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Write messages to a Beanstalkd queue. Introduced in version 4.7.0. #### Common ```yml output: label: "" beanstalkd: address: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml output: label: "" beanstalkd: address: "" # No default (required) max_in_flight: 64 ``` ## [](#fields)Fields ### [](#address)`address` An address to connect to. **Type**: `string` ```yaml # Examples: address: 127.0.0.1:11300 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` --- # Page 135: Migrating from gcp_bigquery to gcp_bigquery_write_api (CDC) **URL**: https://docs.redpanda.com/connect/components/outputs/bigquery_cdc_migration.md --- # Migrating from `gcp_bigquery` to `gcp_bigquery_write_api` (CDC) --- title: Migrating from gcp_bigquery to gcp_bigquery_write_api (CDC) latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/bigquery_cdc_migration page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/bigquery_cdc_migration.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/bigquery_cdc_migration.adoc description: Migrate a CDC pipeline from the load-jobs based gcp_bigquery output to the Storage Write API gcp_bigquery_write_api output. page-git-created-date: "2026-10-10" page-git-modified-date: "2026-10-14" --- The `gcp_bigquery_write_api` output supports BigQuery’s Change Data Capture (CDC) ingestion via the Storage Write API. Compared to the load-jobs based `gcp_bigquery` output, the Storage Write API delivers: - Seconds-scale ingest latency instead of minutes (no batch-and-load cycle). - Per-row UPSERT and DELETE operations without writing intermediate files. - Sequence-number-based out-of-order resolution via `_CHANGE_SEQUENCE_NUMBER`. ## [](#config-translation)Config translation ```yaml # BEFORE: load-jobs based output output: gcp_bigquery: project: my-project dataset: my_dataset table: events format: NEWLINE_DELIMITED_JSON batching: count: 10000 period: 30s # AFTER: Storage Write API CDC mode output: gcp_bigquery_write_api: project: my-project dataset: my_dataset table: events write_mode: upsert_delete change_type: ${! metadata("operation") } change_sequence_number: ${! metadata("scn") } primary_keys: [id] batching: count: 500 period: 1s ``` The `change_type` expression must resolve to `UPSERT` or `DELETE` per message (case-insensitive). `change_sequence_number` is optional but recommended for any pipeline where out-of-order delivery is possible. ## [](#schema-requirements)Schema requirements The destination table must have a `PRIMARY KEY` declared. To add one to an existing table: ```sql ALTER TABLE my_dataset.events ADD PRIMARY KEY (id) NOT ENFORCED; ``` For new tables, use the `auto_create_table` option with `primary_keys`: ```yaml auto_create_table: true schema: - { name: id, type: STRING, mode: REQUIRED } - { name: payload, type: JSON } primary_keys: [id] ``` > 📝 **NOTE** > > Composite primary keys are supported with up to 16 columns. The column order in `primary_keys` is significant for composite keys. > ⚠️ **WARNING** > > `primary_keys` does not add a `PRIMARY KEY` to a pre-existing table. It only applies when `auto_create_table` creates the table. For an existing table without one, run `ALTER TABLE … ADD PRIMARY KEY (…) NOT ENFORCED` first. When `primary_keys` is set and the table already declares a `PRIMARY KEY`, the two must match exactly (same columns, same order). ## [](#snapshot-vs-streaming)Snapshot vs streaming BigQuery’s CDC contract does not permit mixing INSERT (unspecified `_CHANGE_TYPE`) and UPSERT/DELETE rows in the same write. For initial backfills, recommendations are: 1. **UPSERT for everything.** The simplest path: write snapshot rows with `change_type: UPSERT` like any other CDC row. Idempotent; tolerates retries; pays the per-PK merge cost on every snapshot row. Suitable for tables up to ~10M rows. 2. **Separate snapshot pipeline.** Land snapshot rows via a second `gcp_bigquery_write_api` instance with `write_mode: default_stream` into a separate staging table, then `CREATE TABLE … AS SELECT` into the CDC-active table once the snapshot completes. Avoids the merge cost on the snapshot but adds operational complexity. ## [](#operational-differences)Operational differences > ⚠️ **WARNING** > > Tables with active CDC do not support DML statements (`DELETE`, `UPDATE`, `MERGE`). If your existing pipeline relies on post-load DML for cleanup, that pattern will no longer work. Move cleanup to BigQuery scheduled queries against a non-CDC table or pre-process rows in the connector. - BigQuery does not enforce primary-key uniqueness; the producer is responsible. - DELETEs are retained for a two-day window for point-in-time recovery before being permanently dropped. - The `_CHANGE_TYPE` and `_CHANGE_SEQUENCE_NUMBER` pseudo-columns are injected by the connector; do not declare them in `schema`. - CDC ingestion requires the default write stream. The `write_mode: pending_stream` exactly-once mode is not compatible with CDC and is rejected at config parse time. ## [](#related)Related - [`gcp_bigquery_write_api` output reference](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/) - [BigQuery CDC documentation](https://cloud.google.com/bigquery/docs/change-data-capture) --- # Page 136: broker **URL**: https://docs.redpanda.com/connect/components/outputs/broker.md --- # broker --- title: broker latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/broker page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/broker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/broker.adoc description: Routes messages to multiple child outputs using a range of brokering patterns. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A meta-output that routes messages to child outputs using a range of brokering [patterns](#patterns). Unlike regular outputs, `broker` doesn’t send messages anywhere by itself. Instead, it wraps other outputs and controls how messages are delivered across them. Use `broker` to fan out the same message to multiple destinations (for example, publishing events to Kafka while also writing them to a database), or to distribute messages across a pool of outputs for load balancing or throughput scaling. The delivery pattern determines whether each message is written to all outputs or routed to a single output, and whether writes happen in parallel or in sequence. > 📝 **NOTE** > > The name `broker` refers to the brokering delivery pattern, not a Redpanda broker (cluster node). #### Common ```yml output: label: "" broker: pattern: fan_out outputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" broker: copies: 1 pattern: fan_out outputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` [Processors](https://docs.redpanda.com/connect/components/processors/about/) can be listed to apply across individual outputs or all outputs: ```yaml output: broker: pattern: fan_out outputs: - resource: foo - resource: bar # Processors only applied to messages sent to bar. processors: - resource: bar_processor # Processors applied to messages sent to all brokered outputs. processors: - resource: general_processor ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#copies)`copies` The number of copies of each configured output to spawn. **Type**: `int` **Default**: `1` ### [](#outputs)`outputs[]` A list of child outputs to broker. **Type**: `array` ### [](#pattern)`pattern` The brokering pattern to use. **Type**: `string` **Default**: `fan_out` **Options**: `fan_out`, `fan_out_fail_fast`, `fan_out_sequential`, `fan_out_sequential_fail_fast`, `round_robin`, `greedy` ## [](#patterns)Patterns The broker pattern determines how messages are distributed across outputs. Use `fan_out` (the default) when every output should receive every message. Use `round_robin` or `greedy` when you want to distribute messages across outputs for load balancing rather than duplication. The available patterns are: ### [](#fan_out)`fan_out` With the fan out pattern all outputs will be sent every message that passes through Redpanda Connect in parallel. If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates. Sometimes it is useful to disable the back pressure or retries of certain fan out outputs and instead drop messages that have failed or were blocked. In this case you can wrap outputs with a [`drop_on` output](https://docs.redpanda.com/connect/components/outputs/drop_on/). ### [](#fan_out_fail_fast)`fan_out_fail_fast` The same as the `fan_out` pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs. ### [](#fan_out_sequential)`fan_out_sequential` Similar to the fan out pattern except outputs are written to sequentially, meaning an output is only written to once the preceding output has confirmed receipt of the same message. If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates. ### [](#fan_out_sequential_fail_fast)`fan_out_sequential_fail_fast` The same as the `fan_out_sequential` pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs. ### [](#round_robin)`round_robin` With the round robin pattern each message will be assigned a single output following their order. If an output applies back pressure it will block all subsequent messages. If an output fails to send a message then the message will be re-attempted with the next input, and so on. ### [](#greedy)`greedy` The greedy pattern results in higher output throughput at the cost of potentially disproportionate message allocations to those outputs. Each message is sent to a single output, which is determined by allowing outputs to claim messages as soon as they are able to process them. This results in certain faster outputs potentially processing more messages at the cost of slower outputs. --- # Page 137: cache **URL**: https://docs.redpanda.com/connect/components/outputs/cache.md --- # cache --- title: cache latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/cache page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cache.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cache.adoc description: Stores each message in a cache. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Stores each message in a [cache](https://docs.redpanda.com/connect/components/caches/about/). Caches are configured as [resources](https://docs.redpanda.com/connect/components/caches/about/), where there’s a wide variety to choose from. The `target` field must reference a configured cache resource label like follows: ```yaml output: cache: target: foo key: ${!json("document.id")} cache_resources: - label: foo memcached: addresses: - localhost:11211 default_ttl: 60s ``` In order to create a unique `key` value per item you should use function interpolations described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" cache: target: "" # No default (required) key: ${!timestamp_unix_nano()}-${!uuid_v4()} max_in_flight: 64 ``` ### Advanced ```yml output: label: "" cache: target: "" # No default (required) key: ${!timestamp_unix_nano()}-${!uuid_v4()} ttl: "" # No default (optional) max_in_flight: 64 ``` ## [](#fields)Fields ### [](#key)`key` The key to store messages by, function interpolation should be used in order to derive a unique key for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!timestamp_unix_nano()}-${!uuid_v4()}` ```yaml # Examples: key: ${!timestamp_unix_nano()}-${!uuid_v4()} # --- key: ${!json("doc.id")} # --- key: ${!meta("kafka_key")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#target)`target` The target cache to store messages in. **Type**: `string` ### [](#ttl)`ttl` The TTL of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, and those that do not will fall back to their generally configured TTL setting. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.33.0 or later. **Type**: `string` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` --- # Page 138: cassandra **URL**: https://docs.redpanda.com/connect/components/outputs/cassandra.md --- # cassandra --- title: cassandra latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/cassandra page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cassandra.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cassandra.adoc description: Runs a query against a Cassandra database for each message in order to insert data. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Runs a query against a Cassandra database for each message in order to insert data. Query arguments can be set using a bloblang array for the fields using the `args_mapping` field. When populating timestamp columns the value must either be a string in ISO 8601 format (2006-01-02T15:04:05Z07:00), or an integer representing unix time in seconds. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" cassandra: addresses: [] # No default (required) timeout: 600ms reconnect_interval: 60s query: "" # No default (required) args_mapping: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" cassandra: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] password_authenticator: enabled: false username: "" password: "" disable_initial_host_lookup: false max_retries: 3 backoff: initial_interval: 1s max_interval: 5s timeout: 600ms host_selection_policy: local_dc: "" # No default (optional) local_rack: "" # No default (optional) reconnect_interval: 60s exponential_reconnection: max_retries: 0 # No default (required) initial_interval: "" # No default (required) max_interval: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) consistency: QUORUM logged_batch: true max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#examples)Examples ### [](#basic-inserts)Basic Inserts If we were to create a table with some basic columns with `CREATE TABLE foo.bar (id int primary key, content text, created_at timestamp);`, and were processing JSON documents of the form `{"id":"342354354","content":"hello world","timestamp":1605219406}` using logged batches, we could populate our table with the following config: ```yaml output: cassandra: addresses: - localhost:9042 query: 'INSERT INTO foo.bar (id, content, created_at) VALUES (?, ?, ?)' args_mapping: | root = [ this.id, this.content, this.timestamp ] batching: count: 500 period: 1s ``` ### [](#insert-json-documents)Insert JSON Documents The following example inserts JSON documents into the table `footable` of the keyspace `foospace` using INSERT JSON ([https://cassandra.apache.org/doc/latest/cassandra/developing/cql/json.html#insert-json](https://cassandra.apache.org/doc/latest/cassandra/developing/cql/json.html#insert-json)). ```yaml output: cassandra: addresses: - localhost:9042 query: 'INSERT INTO foospace.footable JSON ?' args_mapping: 'root = [ this ]' batching: count: 500 period: 1s ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of Cassandra nodes to connect to. Multiple comma separated addresses can be specified on a single line. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9042" # --- addresses: - "foo:9042" - "bar:9042" # --- addresses: - "foo:9042,bar:9042" ``` ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that can be used to provide arguments to Cassandra queries. The result of the query must be an array containing a matching number of elements to the query arguments. Requires version 3.55.0 or later. **Type**: `string` ### [](#backoff)`backoff` The backoff the Cassandra driver applies between retries of a failed query. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The period to wait before the first retry of a failed query. Each later retry doubles the previous wait, with random jitter of up to half this value, until `backoff.max_interval` is reached. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` The longest wait between retries of a failed query. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#consistency)`consistency` The consistency level to use. **Type**: `string` **Default**: `QUORUM` **Options**: `ANY`, `ONE`, `TWO`, `THREE`, `QUORUM`, `ALL`, `LOCAL_QUORUM`, `EACH_QUORUM`, `LOCAL_ONE` ### [](#disable_initial_host_lookup)`disable_initial_host_lookup` If enabled the driver will not attempt to get host info from the system.peers table. This can speed up queries but will mean that data\_centre, rack and token information will not be available. **Type**: `bool` **Default**: `false` ### [](#exponential_reconnection)`exponential_reconnection` Exponential backoff between the connection attempts that the driver makes each time it connects to a host. It applies on the initial connection and on each `reconnect_interval` attempt to a node marked as DOWN, and replaces the driver’s default of 3 attempts 1s apart. Requires version 4.66.0 or later. **Type**: `object` ### [](#exponential_reconnection-initial_interval)`exponential_reconnection.initial_interval` The base wait between the connection attempts the driver makes each time it connects to a host. The first wait is a random duration up to this value, and each later wait doubles, with random jitter of plus or minus half this value, until `max_interval` is reached. **Type**: `string` ### [](#exponential_reconnection-max_interval)`exponential_reconnection.max_interval` The longest wait between the connection attempts each time the driver connects to a host. **Type**: `string` ### [](#exponential_reconnection-max_retries)`exponential_reconnection.max_retries` The maximum number of connection attempts each time the driver connects to a host. **Type**: `int` ### [](#host_selection_policy)`host_selection_policy` Advanced host selection policy settings for Cassandra clusters, highly recommended in multi-datacenter (DC) deployments. Use these options to optimize query routing in multi-DC and multi-rack deployments. By specifying a local DC and rack, you can use the DC-aware and rack-aware policies to direct queries to the closest nodes, reducing latency and improving fault tolerance. If not set, the default policy is round-robin across all available nodes. Host selection is always token-aware if the token can be calculated from the query. Requires version 4.61.0 or later. **Type**: `object` ```yaml # Examples: host_selection_policy: local_dc: dc-east local_rack: rack1 ``` ### [](#host_selection_policy-local_dc)`host_selection_policy.local_dc` The name of the local datacenter to prioritize for query routing. Enables DC-aware host selection, ensuring queries are sent to nodes within this datacenter whenever possible. Recommended for clusters spanning multiple datacenters to minimize cross-DC traffic. **Type**: `string` ### [](#host_selection_policy-local_rack)`host_selection_policy.local_rack` The name of the local rack to prioritize for query routing. Requires `local_dc` to be set. Enables rack-aware host selection, further optimizing query placement within the specified datacenter. Useful for deployments with multiple racks per datacenter to improve resilience and reduce intra-DC latency. **Type**: `string` ### [](#logged_batch)`logged_batch` If enabled the driver will perform a logged batch. Disabling this prompts unlogged batches to be used instead, which are less efficient but necessary for alternative storages that do not support logged batches. Requires version 4.10.0 or later. **Type**: `bool` **Default**: `true` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on a request. **Type**: `int` **Default**: `3` ### [](#password_authenticator)`password_authenticator` Optional configuration of Cassandra authentication parameters. **Type**: `object` ### [](#password_authenticator-enabled)`password_authenticator.enabled` Whether to use password authentication **Type**: `bool` **Default**: `false` ### [](#password_authenticator-password)`password_authenticator.password` The password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#password_authenticator-username)`password_authenticator.username` The username to authenticate as. **Type**: `string` **Default**: `""` ### [](#query)`query` A query to execute for each message. **Type**: `string` ### [](#reconnect_interval)`reconnect_interval` The interval at which Redpanda Connect attempts to reconnect to Cassandra nodes that are marked as DOWN. This setting helps maintain connectivity in unstable network conditions or during node maintenance. Use Go duration format such as `30s`, `1m`, or `5m`. Setting this too low may create unnecessary connection attempts, while setting it too high may delay recovery from network issues. Requires version 4.66.0 or later. **Type**: `string` **Default**: `60s` ### [](#timeout)`timeout` The client connection timeout. **Type**: `string` **Default**: `600ms` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 139: couchbase **URL**: https://docs.redpanda.com/connect/components/outputs/couchbase.md --- # couchbase --- title: couchbase latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/couchbase page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/couchbase.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/couchbase.adoc description: Performs operations against Couchbase for each message, allowing you to store or delete data. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Perform operations on a [Couchbase database](https://docs.couchbase.com/home/index.html) for each incoming message. This connector can store or delete data in Couchbase based on the message’s content. Introduced in version 4.37.0. #### Common ```yml output: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) id: "" # No default (required) content: "" # No default (optional) operation: upsert max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) collection: "" # No default (optional) scope: "" # No default (optional) transcoder: legacy timeout: 15s id: "" # No default (required) content: "" # No default (optional) ttl: "" # No default (optional) operation: upsert max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance For improved performance, this output sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches), using the `max_in_flight` field. You can configure batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The name of the Couchbase bucket to access. **Type**: `string` ### [](#collection)`collection` The name of the collection within the bucket to access. **Type**: `string` ### [](#content)`content` The document content to update. When inserting, replacing, or upserting documents, you must set a `content` value. **Type**: `string` ### [](#id)`id` The document ID to use. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${! json("id") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` The Couchbase operation to perform. **Type**: `string` **Default**: `upsert` | Option | Summary | | --- | --- | | insert | insert a new document. | | remove | delete a document. | | replace | replace the contents of a document. | | upsert | creates a new document if it does not exist, if it does exist then it updates it. | ### [](#password)`password` A password to authenticate with the Couchbase cluster. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#scope)`scope` Bucket scope. Requires version 4.56.0 or later. **Type**: `string` ### [](#timeout)`timeout` The timeout for Couchbase operations, including connecting to the cluster and waiting for the bucket to become ready. **Type**: `string` **Default**: `15s` ### [](#transcoder)`transcoder` The Couchbase transcoder to use for converting the data format of documents written to and read from Couchbase. **Type**: `string` **Default**: `legacy` | Option | Summary | | --- | --- | | json | JSONTranscoder implements the default transcoding behavior and applies JSON transcoding to all values. This will apply the following behavior to the value: binary ([]byte) → error. default → JSON value, JSON Flags. | | legacy | LegacyTranscoder implements the behavior for a backward-compatible transcoder. This transcoder implements behavior matching that of gocb v1.This will apply the following behavior to the value: binary ([]byte) → binary bytes, Binary expectedFlags. string → string bytes, String expectedFlags. default → JSON value, JSON expectedFlags. | | raw | RawBinaryTranscoder implements passthrough behavior of raw binary data. This transcoder does not apply any serialization. This will apply the following behavior to the value: binary ([]byte) → binary bytes, binary expectedFlags. default → error. | | rawjson | RawJSONTranscoder implements passthrough behavior of JSON data. This transcoder does not apply any serialization. It will forward data across the network without incurring unnecessary parsing costs. This will apply the following behavior to the value: binary ([]byte) → JSON bytes, JSON expectedFlags. string → JSON bytes, JSON expectedFlags. default → error. | | rawstring | RawStringTranscoder implements passthrough behavior of raw string data. This transcoder does not apply any serialization. This will apply the following behavior to the value: string → string bytes, string expectedFlags. default → error. | ### [](#ttl)`ttl` An optional TTL to set for items. Requires version 4.73.0 or later. **Type**: `string` ### [](#url)`url` The Couchbase connection string. **Type**: `string` ```yaml # Examples: url: couchbase://localhost:11210 ``` ### [](#username)`username` A username to authenticate with the Couchbase cluster. **Type**: `string` --- # Page 140: cyborgdb **URL**: https://docs.redpanda.com/connect/components/outputs/cyborgdb.md --- # cyborgdb --- title: cyborgdb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/cyborgdb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cyborgdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cyborgdb.adoc description: Inserts items into a CyborgDB encrypted vector index. page-git-created-date: "2025-10-06" page-git-modified-date: "2026-10-10" --- Inserts items into a CyborgDB encrypted vector index. Introduced in version 4.66.0. This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization. All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure. #### Common ```yaml output: label: "" cyborgdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" host: "" # No default (required) api_key: "" # No default (required) index_name: redpanda-vectors index_key: "" # No default (required) operation: upsert id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` #### Advanced ```yaml output: label: "" cyborgdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) index_name: redpanda-vectors index_key: "" # No default (required) create_if_missing: false operation: upsert id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API key for authenticating with the CyborgDB service. This key identifies your account and provides access to your CyborgDB indexes. Keep this key secure and avoid exposing it in logs or version control. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#create_if_missing)`create_if_missing` Whether to create the index if it doesn’t exist. When enabled, CyborgDB automatically detects the vector dimensions from your data and optimizes the index configuration for performance. This is useful for development and testing environments. **Type**: `bool` **Default**: `false` ### [](#host)`host` The host URL for the CyborgDB instance, including the port number if required. The scheme is optional: if the value does not start with `http://` or `https://`, this output uses `https://`. **Type**: `string` ```yaml # Examples: host: api.cyborg.com # --- host: localhost:8000 ``` ### [](#id)`id` The unique identifier for each vector entry, for example `${! json("id") }`. This ID is used to update existing vectors during upsert operations or to specify which vectors to delete. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#index_key)`index_key` The base64-encoded encryption key for the CyborgDB index. This key must be exactly 32 bytes when decoded from base64. All vector data is encrypted client-side using this key before transmission, ensuring complete data privacy. Store this key securely as it cannot be recovered if lost. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: index_key: your-base64-encoded-32-byte-key ``` ### [](#index_name)`index_name` The name of the CyborgDB index to write vectors to. If the index doesn’t exist and `create_if_missing` is enabled, CyborgDB will create it automatically with optimized settings based on your data. **Type**: `string` **Default**: `redpanda-vectors` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata_mapping)`metadata_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that extracts metadata to associate with the vector entry. The metadata can contain any JSON-serializable data that helps identify or categorize the vector. This data is stored encrypted alongside the vector. **Type**: `string` ```yaml # Examples: metadata_mapping: root = @ # --- metadata_mapping: root = metadata() # --- metadata_mapping: 'root = {"summary": this.summary, "category": this.category}' ``` ### [](#operation)`operation` The operation to perform against the CyborgDB index. Supported operations: - `upsert`: Insert new vectors or update existing ones (requires `vector_mapping`) - `delete`: Remove vectors from the index by `id` **Type**: `string` **Default**: `upsert` **Options**: `upsert`, `delete` ### [](#vector_mapping)`vector_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that extracts the vector from the message. The result must be an array of floating-point numbers representing the vector embeddings. This field is required for the `upsert` operation. **Type**: `string` ```yaml # Examples: vector_mapping: root = this.embeddings_vector # --- vector_mapping: root = [1.2, 0.5, 0.76] ``` --- # Page 141: cypher **URL**: https://docs.redpanda.com/connect/components/outputs/cypher.md --- # cypher --- title: cypher latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/cypher page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cypher.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cypher.adoc description: Writes batches of messages to any database that speaks the Bolt protocol, such as Neo4j or Memgraph, using operations written in the Cypher query language. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Writes a batch of messages to any graph database that supports the [Neo4j and Bolt URI schemes](https://neo4j.com/docs/go-manual/current/connect-advanced/). For each incoming message, the connector can perform operations to store or delete data within the database using the [Cypher query language](https://neo4j.com/docs/cypher-manual/current/introduction/). Introduced in version 4.37.0. #### Common ```yml output: label: "" cypher: uri: "" # No default (required) cypher: "" # No default (required) database_name: "" args_mapping: "" # No default (optional) basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" max_in_flight: 64 ``` #### Advanced ```yml output: label: "" cypher: uri: "" # No default (required) cypher: "" # No default (required) database_name: "" args_mapping: "" # No default (optional) basic_auth: enabled: false username: "" password: "" realm: "" tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` ## [](#examples)Examples ### [](#write-to-neo4j-aura)Write to Neo4j Aura This is an example of how to write to Neo4j Aura ```yaml output: cypher: uri: neo4j+s://example.databases.neo4j.io cypher: | MERGE (product:Product {id: $id}) ON CREATE SET product.name = $product, product.title = $title, product.description = $description, args_mapping: | root = {} root.id = this.product.id root.product = this.product.summary.name root.title = this.product.summary.displayName root.description = this.product.fullDescription basic_auth: enabled: true username: "${NEO4J_USER}" password: "${NEO4J_PASSWORD}" ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` The mapping from the message to the data that is passed in as parameters to the cypher expression. The mapping must return an object. By default, the entire payload is used. **Type**: `string` ```yaml # Examples: args_mapping: root.name = this.displayName # --- args_mapping: 'root = {"orgId": this.org.id, "name": this.user.name}' ``` ### [](#basic_auth)`basic_auth` Basic authentication for the connection to `uri`. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to authenticate with `username`, `password`, and `realm`. When `false`, the connection is made without authentication. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password of the user set in `username`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-realm)`basic_auth.realm` The realm or process for authentication challenges. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The user that the driver authenticates as when it connects to `uri`, with `password` and, optionally, `realm`. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#cypher)`cypher` The Cypher expression to execute against the database. **Type**: `string` ```yaml # Examples: cypher: "MERGE (p:Person {name: $name})" # --- cypher: |- MATCH (o:Organization {id: $orgId}) MATCH (p:Person {name: $name}) MERGE (p)-[:WORKS_FOR]->(o) ``` ### [](#database_name)`database_name` Set the target database against which expressions are evaluated. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#uri)`uri` The connection URI of the database. For more information, see [Neo4j’s documentation](https://neo4j.com/docs/go-manual/current/connect-advanced/). **Type**: `string` ```yaml # Examples: uri: neo4j://demo.neo4jlabs.com # --- uri: neo4j+s://aura.databases.neo4j.io # --- uri: neo4j+ssc://self-signed.demo.neo4jlabs.com # --- uri: bolt://127.0.0.1:7687 # --- uri: bolt+s://core.db.server:7687 # --- uri: bolt+ssc://10.0.0.43 ``` --- # Page 142: discord **URL**: https://docs.redpanda.com/connect/components/outputs/discord.md --- # discord --- title: discord latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/discord page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/discord.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/discord.adoc description: Writes messages to a Discord channel. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Writes messages to a Discord channel. This output POSTs messages to the `/channels/{channel_id}/messages` Discord API endpoint authenticated as a bot using token based authentication. If the format of a message is a JSON object matching the [Discord API message type](https://discord.com/developers/docs/resources/channel#message-object) then it is sent directly, otherwise an object matching the API type is created with the content of the message added as a string. ```yml output: label: "" discord: channel_id: "" # No default (required) bot_token: "" # No default (required) ``` ## [](#fields)Fields ### [](#bot_token)`bot_token` A bot token used for authentication. **Type**: `string` ### [](#channel_id)`channel_id` A discord channel ID to write messages to. **Type**: `string` --- # Page 143: doris_stream_load **URL**: https://docs.redpanda.com/connect/components/outputs/doris_stream_load.md --- # doris_stream_load --- title: doris_stream_load latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/doris_stream_load page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/doris_stream_load.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/doris_stream_load.adoc description: Writes batches of messages into Apache Doris using Stream Load. page-git-created-date: "2026-06-16" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is in beta. Its behavior might change in a future release. Writes batches of messages into Apache Doris using Stream Load. Introduced in version 4.96.0. Each output batch is encoded into a single Doris Stream Load request. The output first contacts a Doris frontend (FE) node and follows the Stream Load redirect to BE (a backend node) before uploading the batch body. A batch is only acknowledged when Doris reports success. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" doris_stream_load: fe_urls: [] database: "" # No default (required) table: "" # No default (required) username: "" # No default (required) password: "" # No default (required) format: json read_json_by_line: true strip_outer_array: false columns: [] label_prefix: redpanda_connect timeout: 30s max_in_flight: 8 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" doris_stream_load: url: "" # No default (optional) fe_urls: [] database: "" # No default (required) table: "" # No default (required) username: "" # No default (required) password: "" # No default (required) query_port: 9030 format: json read_json_by_line: true strip_outer_array: false jsonpaths: "" json_root: "" columns: [] where: "" column_separator: "," line_delimiter: |+ group_commit: "" max_filter_ratio: 0 partitions: [] temporary_partitions: [] skip_lines: 0 empty_field_as_null: false trim_double_quotes: false num_as_string: false fuzzy_parse: false label_prefix: redpanda_connect headers: {} strict_mode: false timeout: 30s max_in_flight: 8 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#column_separator)`column_separator` Column separator used for csv format. **Type**: `string` **Default**: `,` ### [](#columns)`columns[]` Optional Doris columns header. When set the values are joined with commas. **Type**: `array` **Default**: `[]` ### [](#database)`database` Target Doris database. **Type**: `string` ```yaml # Examples: database: test_db ``` ### [](#empty_field_as_null)`empty_field_as_null` Whether Doris should treat empty input fields as NULL. **Type**: `bool` **Default**: `false` ### [](#fe_urls)`fe_urls[]` A list of Doris FE HTTP URLs. The sink will try these FE endpoints in order with per-request failover, and the starting FE is rotated across requests. **Type**: `array` **Default**: `[]` ```yaml # Examples: fe_urls: - "http://fe1:8030" - "http://fe2:8030" ``` ### [](#format)`format` Body format sent to Doris Stream Load. **Type**: `string` **Default**: `json` **Options**: `json`, `csv` ### [](#fuzzy_parse)`fuzzy_parse` Whether Doris should enable fuzzy\_parse for JSON inputs. **Type**: `bool` **Default**: `false` ### [](#group_commit)`group_commit` Optional Doris group\_commit mode. Valid values are sync\_mode, async\_mode, and off\_mode. When omitted Doris uses the server default behavior. **Type**: `string` **Default**: `""` ### [](#headers)`headers` Optional additional static Stream Load headers. Reserved Doris headers configured by this component take precedence. **Type**: `object` **Default**: `{}` ### [](#json_root)`json_root` Optional Doris json\_root header value. **Type**: `string` **Default**: `""` ### [](#jsonpaths)`jsonpaths` Optional Doris jsonpaths header value. **Type**: `string` **Default**: `""` ### [](#label_prefix)`label_prefix` Prefix used when generating Doris labels. **Type**: `string` **Default**: `redpanda_connect` ### [](#line_delimiter)`line_delimiter` Line delimiter used for csv batches. **Type**: `string` **Default**: `"\n"` ### [](#max_filter_ratio)`max_filter_ratio` Optional Doris max\_filter\_ratio header value. **Type**: `float` **Default**: `0` ### [](#max_in_flight)`max_in_flight` Maximum number of parallel in-flight Doris Stream Load requests. **Type**: `int` **Default**: `8` ### [](#num_as_string)`num_as_string` Whether Doris should treat JSON numbers as strings. **Type**: `bool` **Default**: `false` ### [](#partitions)`partitions[]` Optional Doris partitions header. Values are joined with commas. **Type**: `array` **Default**: `[]` ### [](#password)`password` Doris password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#query_port)`query_port` Doris FE MySQL query port, used by ConnectionTest to verify that the target database and table exist. Set to 0 to disable query-port checks. **Type**: `int` **Default**: `9030` ### [](#read_json_by_line)`read_json_by_line` Encode a JSON batch as newline-delimited JSON and set the Doris header read\_json\_by\_line=true. **Type**: `bool` **Default**: `true` ### [](#skip_lines)`skip_lines` Optional Doris skip\_lines header. **Type**: `int` **Default**: `0` ### [](#strict_mode)`strict_mode` Whether to set Doris strict\_mode=true. **Type**: `bool` **Default**: `false` ### [](#strip_outer_array)`strip_outer_array` Encode a JSON batch as a single JSON array and set the Doris header strip\_outer\_array=true. **Type**: `bool` **Default**: `false` ### [](#table)`table` Target Doris table. **Type**: `string` ```yaml # Examples: table: events ``` ### [](#temporary_partitions)`temporary_partitions[]` Optional Doris temporary\_partitions header. Values are joined with commas. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` Timeout for each Doris HTTP request. **Type**: `string` **Default**: `30s` ### [](#trim_double_quotes)`trim_double_quotes` Whether Doris should trim double quotes in CSV/text imports. **Type**: `bool` **Default**: `false` ### [](#url)`url` Backward-compatible single Doris FE HTTP URL, for example [http://fe\_host:8030](http://fe_host:8030). When fe\_urls is provided it takes precedence. **Type**: `string` ```yaml # Examples: url: http://127.0.0.1:8030 ``` ### [](#username)`username` Doris username. **Type**: `string` ```yaml # Examples: username: root ``` ### [](#where)`where` Optional Doris where header for filtering imported rows. **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#json-lines-from-stdin)JSON lines from stdin Read newline-delimited JSON messages and write them into Doris using one Stream Load request per batch. ```yaml input: stdin: scanner: lines: {} output: doris_stream_load: url: http://127.0.0.1:8030 database: test_db table: events username: root password: secret format: json read_json_by_line: true columns: [id, name, ts] ``` --- # Page 144: drop_on **URL**: https://docs.redpanda.com/connect/components/outputs/drop_on.md --- # drop_on --- title: drop_on latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/drop_on page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/drop_on.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/drop_on.adoc description: Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead of being reattempted (or nacked). page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead of being reattempted (or nacked). Regular Redpanda Connect outputs will apply back pressure when downstream services aren’t accessible, and Redpanda Connect retries (or nacks) all messages that fail to be delivered. However, in some circumstances, or for certain output types, we instead might want to relax these mechanisms, which is when this output becomes useful. ```yml output: label: "" drop_on: error: false error_patterns: [] # No default (optional) back_pressure: "" # No default (optional) output: {} # No default (required) ``` ## [](#fields)Fields ### [](#back_pressure)`back_pressure` An optional duration string that determines the maximum length of time to wait for a given message to be accepted by the child output before the message should be dropped instead. The most common reason for an output to block is when waiting for a lost connection to be re-established. Once a message has been dropped due to back pressure all subsequent messages are dropped immediately until the output is ready to process them again. Note that if `error` is set to `false` and this field is specified then messages dropped due to back pressure will return an error response (are nacked or reattempted). **Type**: `string` ```yaml # Examples: back_pressure: 30s # --- back_pressure: 1m ``` ### [](#error)`error` Whether messages should be dropped when the child output returns an error of any type. For example, this could be when an `http_client` output gets a 4XX response code. In order to instead drop only on specific error patterns use the `error_matches` field instead. **Type**: `bool` **Default**: `false` ### [](#error_patterns)`error_patterns[]` A list of regular expressions (re2) where if the child output returns an error that matches any part of any of these patterns the message will be dropped. Requires version 4.27.0 or later. **Type**: `array` ```yaml # Examples: error_patterns: - "and that was really bad$" # --- error_patterns: - "roughly [0-9]+ issues occurred" ``` ### [](#output)`output` A child output to wrap with this drop mechanism. **Type**: `output` ## [](#examples)Examples ### [](#dropping-failed-http-requests)Dropping failed HTTP requests In this example we have a fan\_out broker, where we guarantee delivery to our Kafka output, but drop messages if they fail our secondary HTTP client output. ```yaml output: broker: pattern: fan_out outputs: - kafka: addresses: [ foobar:6379 ] topic: foo - drop_on: error: true output: http_client: url: http://example.com/foo/messages verb: POST ``` ### [](#dropping-from-outputs-that-cannot-connect)Dropping from outputs that cannot connect Most outputs that attempt to establish and long-lived connection will apply back-pressure when the connection is lost. The following example has a websocket output where if it takes longer than 10 seconds to establish a connection, or recover a lost one, pending messages are dropped. ```yaml output: drop_on: back_pressure: 10s output: websocket: url: ws://example.com/foo/messages ``` --- # Page 145: drop **URL**: https://docs.redpanda.com/connect/components/outputs/drop.md --- # drop --- title: drop latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/drop page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/drop.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/drop.adoc description: Drop output reference for silently discarding messages in Redpanda Connect pipelines. page-topic-type: reference personas: streaming_developer, app_developer learning-objective-1: Look up drop output syntax and configuration learning-objective-2: Find examples of drop in conditional routing learning-objective-3: Identify use cases for drop output in pipelines page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Silently discards all messages without error or side effects. The `drop` output is a utility component that drops messages from the [pipeline](#pipeline). Unlike filtering or conditional processors that modify or route messages, `drop` consumes messages and does nothing with them. This is useful for: - **Testing and debugging**: Measure input throughput without output bottlenecks - **Conditional workflows**: Discard messages that don’t meet certain criteria - **Dead letter queue patterns**: Provide a final fallback when all other outputs fail - **Development**: Temporarily disable output while testing pipeline logic Use this reference to: - Look up drop output syntax and configuration - Find examples of drop in conditional routing - Identify use cases for drop output in pipelines ```yaml output: label: "" drop: {} ``` ## [](#performance)Performance The `drop` output has minimal overhead and immediately acknowledges messages. This makes it ideal for performance testing, as it removes output processing time from measurements. ## [](#examples)Examples ### [](#conditional-filtering)Conditional filtering Use `drop` with the [`switch`](https://docs.redpanda.com/connect/components/outputs/switch/) output to conditionally discard messages: ```yaml output: switch: cases: - check: this.type == "error" output: label: error_sink kafka: addresses: ["kafka:9092"] topic: errors - check: this.type == "debug" output: label: drop_debug drop: {} # Don't process debug messages in production - output: label: main_sink kafka: addresses: ["kafka:9092"] topic: events ``` ### [](#dead-letter-queue-pattern)Dead letter queue pattern Use `drop` as a last resort in a [`fallback`](https://docs.redpanda.com/connect/components/outputs/fallback/) chain: ```yaml output: fallback: - kafka: addresses: ["kafka:9092"] topic: primary_topic max_in_flight: 1 - kafka: addresses: ["kafka:9092"] topic: dlq_topic - drop: {} # Last resort: drop if both outputs fail ``` ### [](#testing-input-throughput)Testing input throughput Measure how fast your input can consume data without output bottlenecks: ```yaml input: kafka: addresses: ["kafka:9092"] topics: ["test"] output: drop: {} # Measure input consumption speed without output overhead ``` For more patterns on message routing, filtering, and when to use `drop` vs. other approaches, see the [Message Routing Patterns](https://docs.redpanda.com/connect/cookbooks/message_routing/) cookbook. --- # Page 146: dynamic **URL**: https://docs.redpanda.com/connect/components/outputs/dynamic.md --- # dynamic --- title: dynamic latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/dynamic page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/dynamic.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/dynamic.adoc description: A special broker type where the outputs are identified by unique labels and can be created, changed and removed during runtime via a REST API. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A special broker type where the outputs are identified by unique labels and can be created, changed and removed during runtime via a REST API. #### Common ```yml output: label: "" dynamic: outputs: {} prefix: "" ``` #### Advanced ```yml output: label: "" dynamic: outputs: {} prefix: "" bind_http: false ``` The broker pattern used is always `fan_out`, meaning each message will be delivered to each dynamic output. ## [](#fields)Fields ### [](#bind_http)`bind_http` Whether to register the REST HTTP endpoints (`/outputs`, `/outputs/{id}`, and so on) for creating, updating and removing outputs at runtime. These endpoints are served on the service-wide HTTP server and accept output configuration over HTTP, so they are disabled by default. Enable them only when that server is secured appropriately, for example bound to a trusted interface or protected with `http.basic_auth`. Requires version 4.111.0 or later. **Type**: `bool` **Default**: `false` ### [](#outputs)`outputs` A map of outputs to statically create. **Type**: `object` **Default**: `{}` ### [](#prefix)`prefix` A path prefix for HTTP endpoints that are registered. **Type**: `string` **Default**: `""` ## [](#endpoints)Endpoints ### [](#get-outputs)GET `/outputs` Returns a JSON object detailing all dynamic outputs, providing information such as their current uptime and configuration. ### [](#get-outputsid)GET `/outputs/{id}` Returns the configuration of an output. ### [](#post-outputsid)POST `/outputs/{id}` Creates or updates an output with a configuration provided in the request body (in YAML or JSON format). ### [](#delete-outputsid)DELETE `/outputs/{id}` Stops and removes an output. ### [](#get-outputsiduptime)GET `/outputs/{id}/uptime` Returns the uptime of an output as a duration string (of the form "72h3m0.5s"). --- # Page 147: elasticsearch_v8 **URL**: https://docs.redpanda.com/connect/components/outputs/elasticsearch_v8.md --- # elasticsearch_v8 --- title: elasticsearch_v8 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/elasticsearch_v8 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/elasticsearch_v8.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/elasticsearch_v8.adoc description: Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. page-git-created-date: "2025-03-12" page-git-modified-date: "2026-10-10" --- Publishes messages into an [Elasticsearch index](https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html). If the index does not exist, this output creates it using dynamic mapping. > 📝 **NOTE** > > The `elasticsearch_v8` output is based on the the [go-elasticsearch/v8](https://github.com/elastic/go-elasticsearch?tab=readme-ov-file) library. For full information about breaking changes from previous versions, see [Elastic’s Migrating to 8.0 guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.0.html#breaking_80_rest_api_changes). To help configure your own `elasticsearch_v8` output, this page includes [example pipeline configurations](#example-pipelines). Introduced in version 4.47.0. ### Common ```yml output: label: "" elasticsearch_v8: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 api_key: "" batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" elasticsearch_v8: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" retry_on_conflict: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 api_key: "" basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#set-values-dynamically)Set values dynamically You can use [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to dynamically set values for the [`id`](#id) and [`index`](#index) fields, as well as other fields where [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) are supported. When message batches are sent, interpolations are performed per message. ## [](#performance)Performance For improved performance, this output sends: - Multiple messages in parallel. Adjust the `max_in_flight` field value to tune the maximum number of in-flight messages (or message batches). - Messages as batches. You can configure batches at both input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#action)`action` The action to perform on each document. This field must resolve to one of the following action types: `index`, `update`, `delete`, `create`, or `upsert`. For more information on how the `update` action works, see the `Updating Documents` example. For examples of how to use the `create` and `upsert` actions, see the `Create Documents` and `Upserting Documents` examples. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#api_key)`api_key` An API key to authenticate with. If set, it supersedes basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.96.2 or later. **Type**: `string` **Default**: `""` ### [](#basic_auth)`basic_auth` Configure basic authentication credentials for connecting to Elasticsearch. When enabled, these credentials are sent with each request to authenticate with the cluster. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to send the `username` and `password` credentials with each request. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` The ID for indexed messages. Use [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to dynamically create a unique ID for each message. **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The Elasticsearch index where messages are published. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` The ID of an optional ingest pipeline to preprocess incoming documents before they are indexed. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#retry_on_conflict)`retry_on_conflict` The number of times to retry an update operation when a version conflict occurs. **Type**: `int` **Default**: `0` ### [](#routing)`routing` A custom routing value for each document, which determines the shard it is stored on. When empty, the document ID is used. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` ## [](#example-pipelines)Example pipelines ### Update documents To update documents in the target index, the top level of the request body must include at least one of the following fields: - `doc`: Performs partial updates on a document. - `upsert`: Updates an existing document or inserts a document if it doesn’t exist. - `script`: Performs an update using a scripting language, such as [Elasticsearch’s Painless scripting language](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html). The following examples show how to configure mapping processors with this output to achieve different types of updates. Example 1: Partial document update ```yaml output: processors: # Sets the metadata ID field to the message ID then # performs a partial update on the document. - mapping: | meta id = this.id root.doc = this elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` Example 2: Scripted update ```yaml output: processors: # Sets the metadata ID field to the message ID then # increments the counter field by `1` using a script. - mapping: | meta id = this.id root.script.source = "ctx._source.counter += 1" elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` Example 3: Upsert ```yaml output: processors: # Sets the metadata ID field to the message ID. # If the product with the specified ID exists, update its product_price to 100. # If the document does not exist, insert a new document with the ID set to 1 # and the `product_price` set to 50. - mapping: | meta id = this.id root.doc.product_price = 100 root.upsert.product_price = 50 elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` For more information on the structures and behaviors of `doc`, `upsert`, and `script` fields, see the [Elasticsearch Update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html). ### Index documents from Redpanda Reads messages from a Redpanda cluster and writes them to an Elasticsearch index using a field from the message as the document ID. ```yaml # Reads messages from a Redpanda cluster. input: redpanda: seed_brokers: [localhost:19092] # The address of the Redpanda broker. topics: ["product_code"] # The topic to consume messages from. consumer_group: "rpcn3" # The consumer group ID. processors: # Sets the metadata ID field to the message ID and # sets the root of the message to the message content. - mapping: | meta id = this.id root = this # Writes messages to the specified Elasticsearch index. output: elasticsearch_v8: urls: ['http://localhost:9200'] # The URL of the Elasticsearch server. index: "product_code" # The name of the Elasticsearch index. action: "index" # The action to perform on each document. id: ${! meta("id") } # Sets the document ID to the value of the metadata ID field. ``` ### Index documents from AWS S3 Reads messages from a AWS S3 bucket and writes them to an Elasticsearch index using the S3 key as the ID for the Elasticsearch document. ```yaml # Reads messages from an AWS S3 bucket. input: aws_s3: bucket: "my_bucket" # The name of the S3 bucket. prefix: "prod_inventory/" # A prefix to filter objects in the bucket. scanner: to_the_end: {} # Scans the bucket to the end. # Writes messages to the specified Elasticsearch index. output: elasticsearch_v8: urls: ['http://localhost:9200'] # The URL of the Elasticsearch server. index: "current_prod_inventory" # The name of the Elasticsearch index. action: "index" # The action to perform on each document. id: ${! meta("s3_key") } # Sets the document ID to the S3 key. ``` --- # Page 148: elasticsearch_v9 **URL**: https://docs.redpanda.com/connect/components/outputs/elasticsearch_v9.md --- # elasticsearch_v9 --- title: elasticsearch_v9 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/elasticsearch_v9 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/elasticsearch_v9.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/elasticsearch_v9.adoc description: Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. page-git-created-date: "2026-01-19" page-git-modified-date: "2026-10-10" --- Publishes messages into an Elasticsearch index. If the index does not exist, the output creates one with a dynamic mapping. Both the `id` and `index` fields can be dynamically set using [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. Introduced in version 4.77.0. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in-flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. You can compose a batch at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" elasticsearch_v9: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 api_key: "" batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" elasticsearch_v9: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" retry_on_conflict: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 api_key: "" basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#action)`action` The action to perform on each document. This field must resolve to one of the following action types: `index`, `update`, `delete`, `create`, or `upsert`. For more information on how the `update` action works, see the `Updating Documents` example. For examples of how to use the `create` and `upsert` actions, see the `Create Documents` and `Upserting Documents` examples. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#api_key)`api_key` An API key to authenticate with. If set, it supersedes basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.96.2 or later. **Type**: `string` **Default**: `""` ### [](#basic_auth)`basic_auth` Configure basic authentication credentials for connecting to Elasticsearch. When enabled, these credentials are sent with each request to authenticate with the cluster. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to send the `username` and `password` credentials with each request. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` The ID for indexed messages. Use [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to dynamically create a unique ID for each message. **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The Elasticsearch index where messages are published. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` The ID of an optional ingest pipeline to preprocess incoming documents before they are indexed. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#retry_on_conflict)`retry_on_conflict` The number of times to retry an update operation when a version conflict occurs. **Type**: `int` **Default**: `0` ### [](#routing)`routing` A custom routing value for each document, which determines the shard it is stored on. When empty, the document ID is used. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` ## [](#examples)Examples ### [](#updating-documents)Updating Documents When updating documents, the request body should contain a combination of a `doc`, `upsert`, and/or `script` fields at the top level, this should be done via mapping processors. `doc` updates using a partial document, `script` performs an update using a scripting language such as the built in Painless language, and `upsert` updates an existing document or inserts a new one if it doesn’t exist. For more information on the structures and behaviors of these fields, please see the [Elasticsearch Update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html) ```yaml # Partial document update output: processors: - mapping: | meta id = this.id # Performs a partial update on the document. root.doc = this elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: update # Scripted update output: processors: - mapping: | meta id = this.id # Increments the field "counter" by 1. root.script.source = "ctx._source.counter += 1" elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: update # Upsert output: processors: - mapping: | meta id = this.id # If the product with the ID exists, its price will be updated to 50. # If the product does not exist, a new document with ID 1 and a price # of 100 will be inserted. root.doc.product_price = 50 root.upsert.product_price = 100 elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: update ``` ### [](#indexing-documents-from-redpanda)Indexing documents from Redpanda Here we read messages from a Redpanda cluster and write them to an Elasticsearch index using a field from the message as the ID for the Elasticsearch document. ```yaml input: redpanda: seed_brokers: [localhost:19092] topics: ["things"] consumer_group: "rpcn3" processors: - mapping: | meta id = this.id root = this output: elasticsearch_v9: urls: ['http://localhost:9200'] index: "things" action: "index" id: ${! meta("id") } ``` ### [](#indexing-documents-from-s3)Indexing documents from S3 Here we read messages from a AWS S3 bucket and write them to an Elasticsearch index using the S3 key as the ID for the Elasticsearch document. ```yaml input: aws_s3: bucket: "my-cool-bucket" prefix: "bug-facts/" scanner: to_the_end: {} output: elasticsearch_v9: urls: ['http://localhost:9200'] index: "cool-bug-facts" action: "index" id: ${! meta("s3_key") } ``` ### [](#authenticating-with-an-api-key)Authenticating with an API key Set the `api_key` field to authenticate requests with an Elasticsearch API key. If `api_key` is set, it supersedes the `basic_auth` configuration. ```yaml output: elasticsearch_v9: urls: ['https://localhost:9200'] index: "things" action: "index" id: ${! json("id") } api_key: "${ELASTICSEARCH_API_KEY}" ``` ### [](#create-documents)Create Documents When using the `create` action, a new document will be created if the document ID does not already exist. If the document ID already exists, the operation will fail. ```yaml output: elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! json("id") } action: create ``` ### [](#upserting-documents)Upserting Documents When using the `upsert` action, if the document ID already exists, it will be updated. If the document ID does not exist, a new document will be inserted. The request body should contain the document to be indexed. ```yaml output: processors: - mapping: | meta id = this.id root = this.doc elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: upsert ``` --- # Page 149: fallback **URL**: https://docs.redpanda.com/connect/components/outputs/fallback.md --- # fallback --- title: fallback latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/fallback page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/fallback.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/fallback.adoc description: Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the list is attempted, and so on. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the list is attempted, and so on. Introduced in version 3.58.0. This pattern is useful for triggering events in the case where certain output targets have broken. For example, if you had an output type `http_client` but wished to reroute messages whenever the endpoint becomes unreachable you could use this pattern: ```yaml output: fallback: - http_client: url: http://foo:4195/post/might/become/unreachable retries: 3 retry_period: 1s - http_client: url: http://bar:4196/somewhere/else retries: 3 retry_period: 1s processors: - mapping: 'root = "failed to send this message to foo: " + content()' - file: path: /usr/local/benthos/everything_failed.jsonl ``` ## [](#metadata)Metadata When a given output fails the message routed to the following output will have a metadata value named `fallback_error` containing a string error message outlining the cause of the failure. The content of this string will depend on the particular output and can be used to enrich the message or provide information used to broker the data to an appropriate output using something like a `switch` output. ## [](#batching)Batching When an output within a fallback sequence uses batching, like so: ```yaml output: fallback: - aws_dynamodb: table: foo string_columns: id: ${!json("id")} content: ${!content()} batching: count: 10 period: 1s - file: path: /usr/local/benthos/failed_stuff.jsonl ``` Redpanda Connect makes a best attempt at inferring which specific messages of the batch failed, and only propagates those individual messages to the next fallback tier. However, depending on the output and the error returned it is sometimes not possible to determine the individual messages that failed, in which case the whole batch is passed to the next tier in order to preserve at-least-once delivery guarantees. ```yml output: label: "" fallback: [] ``` --- # Page 150: file **URL**: https://docs.redpanda.com/connect/components/outputs/file.md --- # file --- title: file latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/file page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/file.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/file.adoc description: Writes messages to files on disk based on a chosen codec. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Writes messages to files on disk based on a chosen codec. Messages can be written to different files by using [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) in the path field. However, only one file is ever open at a given time, and therefore when the path changes the previously open file is closed. #### Common ```yml output: label: "" file: path: "" # No default (required) codec: lines ``` #### Advanced ```yml output: label: "" file: path: "" # No default (required) codec: lines ``` ## [](#fields)Fields ### [](#codec)`codec` The way in which the bytes of messages are written to the file. With the `delim:x` codec, each message is followed by the custom delimiter `x`, which can be any character sequence. A delimiter is not added to a message that already ends with it. Requires version 3.33.0 or later. **Type**: `string` **Default**: `lines` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | delim:x | Append each message to the output stream followed by a custom delimiter. | | lines | Append each message to the output stream followed by a line break. | ```yaml # Examples: codec: lines # --- codec: "delim:\t" # --- codec: delim:foobar ``` ### [](#path)`path` The path of the file to write to. The output creates the file and any missing parent directories. A write fails when the path contains a NUL byte. On Windows it also fails when the file name contains `<`, `>`, `:`, `"`, `|`, `?`, `*`, or a control character, and on macOS when the file name contains `:`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.33.0 or later. **Type**: `string` ```yaml # Examples: path: /tmp/data.txt # --- path: /tmp/${! timestamp_unix() }.txt # --- path: /tmp/${! json("document.id") }.json ``` --- # Page 151: gcp_bigquery_write_api **URL**: https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api.md --- # gcp_bigquery_write_api --- title: gcp_bigquery_write_api latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/gcp_bigquery_write_api page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_bigquery_write_api.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_bigquery_write_api.adoc description: Streams data into BigQuery using the Storage Write API. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data into BigQuery using the Storage Write API. Introduced in version 4.90.0. Writes messages to a BigQuery table using the Storage Write API. This provides higher throughput and lower latency than the legacy streaming API or load jobs. Messages can be formatted as JSON (default) or raw protobuf bytes. When using JSON format the component automatically fetches the table schema and converts each message to the corresponding proto representation. > ⚠️ **WARNING** > > The proto3 JSON mapping encodes int64 and uint64 values as strings. JSON messages with integer fields must use string values (for example, `"age": "30"` not `"age": 30`). Otherwise the write will fail with an unmarshalling error. When batching is enabled the table name is resolved from the first message in each batch. All messages in the same batch are written to that table. #### Common ```yml output: label: "" gcp_bigquery_write_api: project: "" dataset: "" # No default (required) table: "" # No default (required) message_format: json change_type: "" # No default (optional) change_sequence_number: "" # No default (optional) primary_keys: [] # No default (optional) max_in_flight: 4 batching: count: 0 byte_size: 0 period: "" check: "" credentials_json: "" ``` #### Advanced ```yml output: label: "" gcp_bigquery_write_api: project: "" dataset: "" # No default (required) table: "" # No default (required) message_format: json write_mode: default_stream change_type: "" # No default (optional) change_sequence_number: "" # No default (optional) primary_keys: [] # No default (optional) auto_create_table: false schema: [] time_partitioning: type: "" # No default (optional) field: "" expiration: 0s require_filter: false clustering: [] max_in_flight: 4 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) credentials_json: "" target_principal: "" delegates: [] stream_idle_timeout: 5m stream_sweep_interval: 1m max_cached_streams: 1024 schema_resolve_timeout: 15s schema_evolution_timeout: 30s endpoint: http: "" grpc: "" ``` ## [](#fields)Fields ### [](#auto_create_table)`auto_create_table` If true and the target table does not exist, the output creates it using the configured `schema`, `time_partitioning`, and `clustering`. AlreadyExists errors from concurrent creators are treated as success. When the table name is interpolated, every auto-created table receives the same schema and partition/clustering settings. Requires version 4.94.0 or later. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#change_sequence_number)`change_sequence_number` Optional Bloblang expression resolving to the `_CHANGE_SEQUENCE_NUMBER` pseudo-column value. Format: 1 to 4 sections of 1 to 16 hexadecimal characters each, separated by `/`. Example: `${! metadata("scn") }` or `${! "0/0/0/0" }`. When unset, BigQuery resolves ordering by arrival time. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.96.0 or later. **Type**: `string` ### [](#change_type)`change_type` Bloblang expression resolving to the `_CHANGE_TYPE` pseudo-column value for each row. Must resolve to `UPSERT` or `DELETE` (case-insensitive). Required when `write_mode` is `upsert` or `upsert_delete`. Example: `${! metadata("operation") }`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.96.0 or later. **Type**: `string` ### [](#clustering)`clustering[]` Optional clustering columns (up to 4) applied during `auto_create_table`. All names must appear in `schema`. Requires version 4.94.0 or later. **Type**: `array` **Default**: `[]` ### [](#credentials_json)`credentials_json` An optional JSON string containing GCP credentials. If empty, credentials are loaded from the environment. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#dataset)`dataset` The BigQuery dataset ID. **Type**: `string` ### [](#delegates)`delegates[]` Optional delegation chain for chained service account impersonation. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain. **Type**: `array` **Default**: `[]` ### [](#endpoint)`endpoint` Optional endpoint overrides for the BigQuery and Storage Write API clients. **Type**: `object` ### [](#endpoint-grpc)`endpoint.grpc` Override the BigQuery Storage gRPC endpoint. Useful for local emulators. **Type**: `string` **Default**: `""` ### [](#endpoint-http)`endpoint.http` Override the BigQuery HTTP endpoint. Useful for local emulators. **Type**: `string` **Default**: `""` ### [](#max_cached_streams)`max_cached_streams` Soft cap on the number of cached streams. When the cache exceeds this size, the least-recently-used stream is evicted. Set to 0 for unlimited (rely on idle-timeout sweeping only). Relevant when the table field uses interpolation to route to many tables. Requires version 4.93.0 or later. **Type**: `int` **Default**: `1024` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `4` ### [](#message_format)`message_format` The format of input messages. Use 'json' to have the component convert JSON to proto automatically. Use 'protobuf' to supply raw proto-encoded bytes. **Type**: `string` **Default**: `json` **Options**: `json`, `protobuf` ### [](#primary_keys)`primary_keys[]` Optional list of primary-key column names. Required when `auto_create_table` is true and `write_mode` is `upsert` or `upsert_delete`. A pre-existing table must already declare its PRIMARY KEY: this field cannot add one; when both are set they must match exactly (same columns, same order). Up to 16 columns; composite keys are supported in the same order they are listed. Requires version 4.96.0 or later. **Type**: `array` ### [](#project)`project` The GCP project ID. If empty, the project is auto-detected from the environment. **Type**: `string` **Default**: `""` ### [](#schema)`schema[]` Column definitions used by `auto_create_table`. Required when `auto_create_table` is true. Requires version 4.94.0 or later. **Type**: `array` **Default**: `[]` ### [](#schema-fields)`schema[].fields[]` For RECORD columns, the list of nested fields. Same shape as the top-level schema list. **Type**: `array` ### [](#schema-mode)`schema[].mode` Column mode: NULLABLE (default), REQUIRED, or REPEATED. **Type**: `string` **Default**: `NULLABLE` ### [](#schema-name)`schema[].name` Column name. **Type**: `string` ### [](#schema-type)`schema[].type` BigQuery column type (STRING, BYTES, INTEGER/INT64, FLOAT/FLOAT64, NUMERIC, BIGNUMERIC, BOOLEAN/BOOL, TIMESTAMP, DATE, TIME, DATETIME, GEOGRAPHY, JSON, RECORD). **Type**: `string` ### [](#schema_evolution_timeout)`schema_evolution_timeout` Total time budget for a single schema evolution attempt (Metadata + Update across all CAS retries on HTTP 412). Bounds how long the WriteBatch retry loop can be starved by a wedged backend. Requires version 4.93.0 or later. **Type**: `string` **Default**: `30s` ### [](#schema_resolve_timeout)`schema_resolve_timeout` How long a single BigQuery table-metadata fetch can run before being aborted. Coalesced concurrent resolves share one fetch, so this bounds the time a wedged backend can stall every batch routing to the same table. On the auto\_create\_table path the budget covers Metadata→Create→Metadata, so it needs to absorb transient backend slowness on top of the metadata fetch itself. Requires version 4.93.0 or later. **Type**: `string` **Default**: `15s` ### [](#stream_idle_timeout)`stream_idle_timeout` How long a cached stream can remain unused before being closed. Relevant when the table field uses interpolation to route to many tables. **Type**: `string` **Default**: `5m` ### [](#stream_sweep_interval)`stream_sweep_interval` How often to check for idle streams to close. **Type**: `string` **Default**: `1m` ### [](#table)`table` The BigQuery table ID. Supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When batching, resolved from the first message in each batch. **Type**: `string` ### [](#target_principal)`target_principal` Service account email to impersonate. When set, the output obtains tokens acting as this service account. Requires the caller to have roles/iam.serviceAccountTokenCreator on the target. **Type**: `string` **Default**: `""` ### [](#time_partitioning)`time_partitioning` Optional time-partitioning settings applied during `auto_create_table`. Setting `type` is the trigger: when omitted, the block is treated as absent. Requires version 4.94.0 or later. **Type**: `object` ### [](#time_partitioning-expiration)`time_partitioning.expiration` Optional partition expiration. Zero means no expiration. **Type**: `string` **Default**: `0s` ### [](#time_partitioning-field)`time_partitioning.field` Column to partition on. Must be of type DATE, TIMESTAMP, or DATETIME. If empty, the table uses ingestion-time partitioning (`_PARTITIONTIME`). **Type**: `string` **Default**: `""` ### [](#time_partitioning-require_filter)`time_partitioning.require_filter` If true, queries against the table must filter on the partition column. **Type**: `bool` **Default**: `false` ### [](#time_partitioning-type)`time_partitioning.type` Partitioning granularity. **Type**: `string` **Options**: `DAY`, `HOUR`, `MONTH`, `YEAR` ### [](#write_mode)`write_mode` How the output writes to BigQuery. `default_stream` uses the multiplexed default stream (at-least-once, lowest latency). `pending_stream` allocates a per-batch pending stream that commits atomically, providing exactly-once semantics within a single committed batch. `upsert` writes UPSERT-only rows to a BigQuery CDC-enabled table; the target table must have a PRIMARY KEY. `upsert_delete` allows both UPSERT and DELETE rows. Both CDC modes use the default stream as required by BigQuery. Requires version 4.94.0 or later. **Type**: `string` **Default**: `default_stream` **Options**: `default_stream`, `pending_stream`, `upsert`, `upsert_delete` --- # Page 152: gcp_bigquery **URL**: https://docs.redpanda.com/connect/components/outputs/gcp_bigquery.md --- # gcp_bigquery --- title: gcp_bigquery latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/gcp_bigquery page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_bigquery.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_bigquery.adoc description: Sends messages as new rows to a Google Cloud BigQuery table. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Inserts message data as new rows in a Google Cloud BigQuery table. Introduced in version 3.55.0. #### Common ```yml output: label: "" gcp_bigquery: project: "" job_project: "" dataset: "" # No default (required) table: "" # No default (required) format: NEWLINE_DELIMITED_JSON max_in_flight: 64 job_labels: {} credentials_json: "" csv: header: [] field_delimiter: "," batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" gcp_bigquery: project: "" job_project: "" dataset: "" # No default (required) table: "" # No default (required) format: NEWLINE_DELIMITED_JSON max_in_flight: 64 write_disposition: WRITE_APPEND create_disposition: CREATE_IF_NEEDED ignore_unknown_values: false max_bad_records: 0 auto_detect: false job_labels: {} credentials_json: "" csv: header: [] field_delimiter: "," allow_jagged_rows: false allow_quoted_newlines: false encoding: UTF-8 skip_leading_rows: 1 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a [shared credentials file](https://docs.redpanda.com/connect/guides/cloud/gcp/) when connecting to GCP services. ## [](#format)Format The `gcp_bigquery` output currently supports only `NEWLINE_DELIMITED_JSON`, `CSV` and `PARQUET` formats. To learn more about how to use BigQuery with these formats, see the following documentation: - [`NEWLINE_DELIMITED_JSON`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json) - [`CSV`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv) - [`PARQUET`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet) ### [](#newline-delimited-json)Newline-delimited JSON Each JSON message may contain multiple elements separated by newlines. For example, a single message containing: ```json {"key": "1"} {"key": "2"} ``` Is equivalent to two separate messages: ```json {"key": "1"} ``` And: ```json {"key": "2"} ``` The same is true for the CSV format. ### [](#csv)CSV When the field `csv.header` is specified for the `CSV` format, a header row is inserted as the first line of each message batch. If this field is not provided, then the first message of each message batch must include a header line. ### [](#parquet)Parquet Each message sent to this output must be a Parquet file. You can use the [`parquet_encode` processor](https://docs.redpanda.com/connect/components/processors/parquet_encode/) to convert message data into the correct format. For example: ```yaml input: generate: mapping: | root = { "foo": random_int(), "bar": uuid_v4(), "time": now(), } interval: 0 count: 1000 batch_size: 1000 pipeline: processors: - parquet_encode: schema: - name: foo type: INT64 - name: bar type: UTF8 - name: time type: UTF8 default_compression: zstd output: gcp_bigquery: project: "${PROJECT}" dataset: "my_bq_dataset" table: "redpanda_connect_ingest" format: PARQUET ``` ## [](#performance)Performance The `gcp_bigquery` output benefits from sending multiple messages in parallel for improved performance. You can tune the maximum number of in-flight messages (or message batches) with the field `max_in_flight`. This output also sends messages as a batch for improved performance. Redpanda Connect can form batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#auto_detect)`auto_detect` Whether this component automatically infers the options and schema for `CSV` and `NEWLINE_DELIMITED_JSON` sources. If this value is set to `false` and the destination table doesn’t exist, the output throws an insertion error as it is unable to insert data. > ⚠️ **CAUTION** > > This field delegates schema detection to the GCP BigQuery service. For the `CSV` format, values like `no` may be treated as booleans. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#create_disposition)`create_disposition` Specifies the circumstances under which a destination table is created. - Use `CREATE_IF_NEEDED` to create the destination table if it does not already exist. Tables are created atomically on successful completion of a job. - Use `CREATE_NEVER` if the destination table must already exist. Tables are not created automatically. **Type**: `string` **Default**: `CREATE_IF_NEEDED` **Options**: `CREATE_IF_NEEDED`, `CREATE_NEVER` ### [](#credentials_json)`credentials_json` Sets the [Google Service Account Credentials JSON](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account) (optional). > ⚠️ **WARNING** > > When using [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to populate this field, wrap the function in single quotes, not double quotes. For example, use `'$\{secrets.GCP_CREDENTIALS_JSON}'` instead of `"$\{secrets.GCP_CREDENTIALS_JSON}"`. Double quotes cause JSON parsing errors because the credentials already contain JSON content. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#csv-2)`csv` Specify how CSV data is interpreted. **Type**: `object` ### [](#csv-allow_jagged_rows)`csv.allow_jagged_rows` Set to `true` to tolerate missing trailing optional columns in CSV data. Missing values are treated as nulls. **Type**: `bool` **Default**: `false` ### [](#csv-allow_quoted_newlines)`csv.allow_quoted_newlines` Whether quoted data sections containing new lines are allowed when reading CSV data. **Type**: `bool` **Default**: `false` ### [](#csv-encoding)`csv.encoding` The character encoding of CSV data. **Type**: `string` **Default**: `UTF-8` **Options**: `UTF-8`, `ISO-8859-1` ### [](#csv-field_delimiter)`csv.field_delimiter` The separator for fields in a CSV file. The output uses this value when reading or exporting data. **Type**: `string` **Default**: `,` ### [](#csv-header)`csv.header[]` A list of values to use as the header for each batch of messages. If not specified, the first line of each message is used as the header. **Type**: `array` **Default**: `[]` ### [](#csv-skip_leading_rows)`csv.skip_leading_rows` The number of rows at the top of a CSV file that BigQuery will skip when reading data. The default value is `1`, which allows Redpanda Connect to add the specified header in the first line of each batch sent to BigQuery. **Type**: `int` **Default**: `1` ### [](#dataset)`dataset` The BigQuery Dataset ID. **Type**: `string` ### [](#format-2)`format` The format of each incoming message. **Type**: `string` **Default**: `NEWLINE_DELIMITED_JSON` **Options**: `NEWLINE_DELIMITED_JSON`, `CSV`, `PARQUET` ### [](#ignore_unknown_values)`ignore_unknown_values` Set this value to `true` to tolerate values that do not match the schema. Unknown values are ignored: - For the `CSV` format, extra values at the end of a line are ignored. - For the `NEWLINE_DELIMITED_JSON` format, values that do not match any column name are ignored. By default, this value is set to `false`, and records containing unknown values are treated as bad records. Use the `max_bad_records` field to customize how bad records are handled. **Type**: `bool` **Default**: `false` ### [](#job_labels)`job_labels` A map of labels to add to the load job. Requires version 4.1.0 or later. **Type**: `object` **Default**: `{}` ### [](#job_project)`job_project` Specify the project ID in which jobs are executed. If not set, the `project` value is used. Requires version 4.35.0 or later. **Type**: `string` **Default**: `""` ### [](#max_bad_records)`max_bad_records` The maximum number of bad records that BigQuery ignores when reading data. This includes records with unknown values when `ignore_unknown_values` is `false`. If the number of bad records exceeds this value, the load job fails. **Type**: `int` **Default**: `0` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to write to BigQuery in parallel. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#project)`project` Specify the project ID of the dataset to insert data into. If not set, the project ID is inferred from the project linked to the service account or read from the `GOOGLE_CLOUD_PROJECT` environment variable. **Type**: `string` **Default**: `""` ### [](#table)`table` The table to insert messages into. **Type**: `string` ### [](#write_disposition)`write_disposition` Specifies how existing data in a destination table is treated. **Type**: `string` **Default**: `WRITE_APPEND` **Options**: `WRITE_APPEND`, `WRITE_EMPTY`, `WRITE_TRUNCATE` --- # Page 153: gcp_cloud_storage **URL**: https://docs.redpanda.com/connect/components/outputs/gcp_cloud_storage.md --- # gcp_cloud_storage --- title: gcp_cloud_storage latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/gcp_cloud_storage page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_cloud_storage.adoc description: Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the path field. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the `path` field. Introduced in version 3.43.0. In order to have a different path for each object you should use function interpolations described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. ## [](#metadata)Metadata Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](https://docs.redpanda.com/connect/configuration/metadata/). ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in [Google Cloud Platform](https://docs.redpanda.com/connect/guides/cloud/gcp/). ## [](#batching)Batching It’s common to want to upload messages to Google Cloud Storage as batched archives, the easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](https://docs.redpanda.com/connect/components/processors/archive/) and/or [`compress`](https://docs.redpanda.com/connect/components/processors/compress/) processor. For example, if we wished to upload messages as a .tar.gz archive of documents we could achieve that with the following config: ```yaml output: gcp_cloud_storage: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz batching: count: 100 period: 10s processors: - archive: format: tar - compress: algorithm: gzip ``` Alternatively, if we wished to upload JSON documents as a single large document containing an array of objects we can do that with: ```yaml output: gcp_cloud_storage: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.json batching: count: 100 processors: - archive: format: json_array ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" gcp_cloud_storage: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt content_type: application/octet-stream collision_mode: overwrite timeout: 3s credentials_json: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" gcp_cloud_storage: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt content_type: application/octet-stream content_encoding: "" collision_mode: overwrite chunk_size: 16777216 timeout: 3s credentials_json: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The bucket to upload messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#chunk_size)`chunk_size` An optional chunk size which controls the maximum number of bytes of the object that the Writer will attempt to send to the server in a single request. If ChunkSize is set to zero, chunking will be disabled. **Type**: `int` **Default**: `16777216` ### [](#collision_mode)`collision_mode` Determines how file path collisions should be dealt with. Options are "overwrite", which replaces the existing file with the new one, "append", which appends the message bytes to the original file, "error-if-exists", which returns an error and rejects the message if the file exists, and "ignore", does not modify the original file and drops the message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.53.0 or later. **Type**: `string` **Default**: `overwrite` **Options**: `overwrite`, `append`, `error-if-exists`, `ignore` ### [](#content_encoding)`content_encoding` An optional value for the `contentEncoding` metadata of each Cloud Storage object, such as `gzip` for objects that are stored compressed. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The MIME type stored in the `contentType` metadata of each Cloud Storage object, which is returned as the `Content-Type` header when the object is downloaded. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials_json)`credentials_json` The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#timeout)`timeout` The maximum period to wait for a message batch to be written to Cloud Storage, including any collision checks and appends. When the timeout is reached, the write is abandoned and reattempted. Requires version 4.17.0 or later. **Type**: `string` **Default**: `3s` ```yaml # Examples: timeout: 1s # --- timeout: 500ms ``` --- # Page 154: gcp_pubsub **URL**: https://docs.redpanda.com/connect/components/outputs/gcp_pubsub.md --- # gcp_pubsub --- title: gcp_pubsub latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/gcp_pubsub page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_pubsub.adoc description: Sends messages to a GCP Cloud Pub/Sub topic. Metadata from messages are sent as attributes. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to a GCP Cloud Pub/Sub topic. [Metadata](https://docs.redpanda.com/connect/configuration/metadata/) from messages are sent as attributes. For information on how to set up credentials, see [this guide](https://cloud.google.com/docs/authentication/production). ## [](#troubleshooting)Troubleshooting If you’re consistently seeing `Failed to send message to gcp_pubsub: context deadline exceeded` error logs without any further information it is possible that you are encountering [https://github.com/redpanda-data/connect/issues/1042](https://github.com/redpanda-data/connect/issues/1042), which occurs when metadata values contain characters that are not valid utf-8. This can frequently occur when consuming from Kafka as the key metadata field may be populated with an arbitrary binary value, but this issue is not exclusive to Kafka. If you are blocked by this issue then a work around is to delete either the specific problematic keys: ```yaml pipeline: processors: - mapping: | meta kafka_key = deleted() ``` Or delete all keys with: ```yaml pipeline: processors: - mapping: meta = deleted() ``` ### Common ```yml output: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" topic: "" # No default (required) endpoint: "" max_in_flight: 64 count_threshold: 100 delay_threshold: 10ms byte_threshold: 1000000 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" topic: "" # No default (required) endpoint: "" ordering_key: "" # No default (optional) max_in_flight: 64 count_threshold: 100 delay_threshold: 10ms byte_threshold: 1000000 publish_timeout: 1m0s validate_topic: true metadata: exclude_prefixes: [] flow_control: max_outstanding_bytes: -1 max_outstanding_messages: 1000 limit_exceeded_behavior: block batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configures a batching policy on this output. While the PubSub client maintains its own internal buffering mechanism, preparing larger batches of messages can further trade-off some latency for throughput. Requires version 4.16.0 or later. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#byte_threshold)`byte_threshold` Publish a batch when its size in bytes reaches this value. Requires version 4.16.0 or later. **Type**: `int` **Default**: `1000000` ### [](#count_threshold)`count_threshold` Publish a pubsub buffer when it has this many messages Requires version 4.16.0 or later. **Type**: `int` **Default**: `100` ### [](#credentials_json)`credentials_json` The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#delay_threshold)`delay_threshold` Publish a non-empty pubsub buffer after this delay has passed. Requires version 4.16.0 or later. **Type**: `string` **Default**: `10ms` ### [](#endpoint)`endpoint` An optional endpoint that overrides the default of `pubsub.googleapis.com:443`. Use this field to connect to a region-specific Pub/Sub endpoint. For a list of valid values, see [Pub/Sub regional endpoints](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_regional_endpoints). Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: endpoint: us-central1-pubsub.googleapis.com:443 # --- endpoint: us-west3-pubsub.googleapis.com:443 ``` ### [](#flow_control)`flow_control` For a given topic, configures the PubSub client’s internal buffer for messages to be published. Requires version 4.12.0 or later. **Type**: `object` ### [](#flow_control-limit_exceeded_behavior)`flow_control.limit_exceeded_behavior` Configures the behavior when trying to publish additional messages while the flow controller is full. The available options are block (default), ignore (disable), and signal\_error (publish results will return an error). **Type**: `string` **Default**: `block` **Options**: `ignore`, `block`, `signal_error` ### [](#flow_control-max_outstanding_bytes)`flow_control.max_outstanding_bytes` Maximum size of buffered messages to be published. If less than or equal to zero, this is disabled. **Type**: `int` **Default**: `-1` ### [](#flow_control-max_outstanding_messages)`flow_control.max_outstanding_messages` Maximum number of buffered messages to be published. If less than or equal to zero, this is disabled. **Type**: `int` **Default**: `1000` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increasing this may improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as attributes, all are sent by default. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#ordering_key)`ordering_key` The ordering key to use for publishing messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#project)`project` The project ID of the topic to publish to. **Type**: `string` ### [](#publish_timeout)`publish_timeout` The maximum length of time to wait before abandoning a publish attempt for a message. **Type**: `string` **Default**: `1m0s` ```yaml # Examples: publish_timeout: 10s # --- publish_timeout: 5m # --- publish_timeout: 60m ``` ### [](#topic)`topic` The topic to publish to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#validate_topic)`validate_topic` Whether to validate the existence of the topic before publishing. If set to false and the topic does not exist, messages will be lost. Requires version 4.59.0 or later. **Type**: `bool` **Default**: `true` --- # Page 155: hdfs **URL**: https://docs.redpanda.com/connect/components/outputs/hdfs.md --- # hdfs --- title: hdfs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/hdfs page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/hdfs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/hdfs.adoc description: Sends message parts as files to a HDFS directory. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends message parts as files to a HDFS directory. Each file is written with the path specified with the 'path' field, in order to have a different path for each object you should use function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" hdfs: hosts: [] # No default (required) user: "" directory: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" hdfs: hosts: [] # No default (required) user: "" directory: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#directory)`directory` A directory to store message files within. If the directory does not exist it will be created. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#hosts)`hosts[]` A list of target host addresses to connect to. **Type**: `array` ```yaml # Examples: hosts: localhost:9000 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path to upload messages as. Use [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to generate unique file paths. **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ### [](#user)`user` A user ID to connect as. **Type**: `string` **Default**: `""` --- # Page 156: http_client **URL**: https://docs.redpanda.com/connect/components/outputs/http_client.md --- # http_client --- title: http_client latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/http_client page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/http_client.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/http_client.adoc description: Sends messages to an HTTP server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to a HTTP server. #### Common ```yml output: label: "" http_client: url: "" # No default (required) verb: POST headers: {} rate_limit: "" # No default (optional) timeout: 5s max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" http_client: url: "" # No default (required) verb: POST headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false batch_as_multipart: false propagate_response: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) multipart: [] ``` ## [](#message-sends)Message sends The body of the request sent to the HTTP server is the raw contents of the message payload. If the message has multiple parts (is a batch), the request is sent according to [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). To disable this behavior, set the [`batch_as_multipart`](#batch_as_multipart) field to `false`. When message retries are exhausted, this output rejects a message. Typically, a pipeline then continues attempts to send the message until it succeeds, whilst applying back pressure. ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#propagate-responses)Propagate responses To propagate HTTP responses back to the input source, set the [`propagate_response`](#propagate_response) field to `true` . This feature is only available for inputs that support [synchronous responses](https://docs.redpanda.com/connect/guides/sync_responses/). ## [](#performance)Performance For improved performance, this output sends: - Multiple messages in parallel. Adjust the `max_in_flight` field value to tune the maximum number of in-flight messages (or message batches). - Messages as batches. You can configure batches at both input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure and trigger retries with an increasing backoff period between attempts. **Type**: `array` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#batch_as_multipart)`batch_as_multipart` When set to `true`, sends all messages in a batch as a single request using [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). When set to `false`, sends messages in a batch as individual requests. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. Requires version 4.47.0 or later. **Type**: `bool` **Default**: `false` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure where retries should not be attempted. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `array` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, `""` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting synchronous response messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. This field is only applicable when `propagate_response` is set to `true`. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether to transparently follow redirects, that is, responses with HTTP status codes in the 300-399 range. If set to `false`, the response message includes the body, status, and headers from the redirect response, and the component does not make a request to the URL specified in the `Location` header. Requires version 4.39.0 or later. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys should be added to the HTTP request as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#multipart)`multipart[]` EXPERIMENTAL: Create explicit multipart HTTP requests by specifying an array of parts to add to a request. Each part consists of content headers and a data field, which can be populated dynamically. If populated, this field overrides the default request creation behavior. Requires version 3.63.0 or later. **Type**: `array` **Default**: `[]` ### [](#multipart-body)`multipart[].body` The body of the individual message part. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: body: ${! this.data.part1 } ``` ### [](#multipart-content_disposition)`multipart[].content_disposition` The content disposition of the individual message part. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: content_disposition: form-data; name="bin"; filename='${! @AttachmentName } ``` ### [](#multipart-content_type)`multipart[].content_type` The content type of the individual message part. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: content_type: application/bin ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. Requires version 4.21.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). Requires version 3.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#propagate_response)`propagate_response` Whether to [propagate server responses back](https://docs.redpanda.com/connect/guides/sync_responses/) to the input. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` An HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](https://docs.redpanda.com/connect/components/rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The period to wait between failed requests before retrying. For status codes listed in `backoff_on`, this is the starting period, which increases with each attempt up to `max_retry_backoff`. **Type**: `string` **Default**: `1s` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in the `backoff_on` or `drop_on` fields, regardless of this field. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `POST` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 157: http_server **URL**: https://docs.redpanda.com/connect/components/outputs/http_server.md --- # http_server --- title: http_server latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/http_server page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/http_server.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/http_server.adoc description: Sets up an HTTP server that will send messages over HTTP(S) GET requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sets up an HTTP server that will send messages over HTTP(S) GET requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. Sets up an HTTP server that will send messages over HTTP(S) GET requests. If the `address` config field is left blank the [service-wide HTTP server](https://docs.redpanda.com/connect/components/http/about/) will be used. Three endpoints will be registered at the paths specified by the fields `path`, `stream_path` and `ws_path`. Which allow you to consume a single message batch, a continuous stream of line delimited messages, or a websocket of messages for each request respectively. When messages are batched the `path` endpoint encodes the batch according to [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). This behavior can be overridden by [archiving your batches](https://docs.redpanda.com/connect/configuration/batching/#post-batch-processing). Please note, messages are considered delivered as soon as the data is written to the client. There is no concept of at least once delivery on this output. > ⚠️ **CAUTION: Endpoint caveats** > > Endpoint caveats > > Components within a Redpanda Connect config will register their respective endpoints in a non-deterministic order. This means that establishing precedence of endpoints that are registered via multiple `http_server` inputs or outputs (either within brokers or from cohabiting streams) is not possible in a predictable way. > > This ambiguity makes it difficult to ensure that paths which are both a subset of a path registered by a separate component, and end in a slash (`/`) and will therefore match against all extensions of that path, do not prevent the more specific path from matching against requests. > > It is therefore recommended that you ensure paths of separate components do not collide unless they are explicitly non-competing. > > For example, if you were to deploy two separate `http_server` inputs, one with a path `/foo/` and the other with a path `/foo/bar`, it would not be possible to ensure that the path `/foo/` does not swallow requests made to `/foo/bar`. #### Common ```yml output: label: "" http_server: address: "" path: /get stream_path: /get/stream ws_path: /get/ws allowed_verbs: - "GET" ``` #### Advanced ```yml output: label: "" http_server: address: "" path: /get stream_path: /get/stream ws_path: /get/ws allowed_verbs: - "GET" timeout: 5s cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] ``` ## [](#fields)Fields ### [](#address)`address` An alternative address to host from. If left empty the service wide address is used. **Type**: `string` **Default**: `""` ### [](#allowed_verbs)`allowed_verbs[]` An array of verbs that are allowed for the `path` and `stream_path` HTTP endpoint. **Type**: `array` **Default**: ```yaml - "GET" ``` ### [](#cert_file)`cert_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#cors)`cors` Adds Cross-Origin Resource Sharing headers. Only valid with a custom `address`. Requires version 3.63.0 or later. **Type**: `object` ### [](#cors-allowed_origins)`cors.allowed_origins[]` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#key_file)`key_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#path)`path` The path from which discrete messages can be consumed. **Type**: `string` **Default**: `/get` ### [](#stream_path)`stream_path` The path from which a continuous stream of messages can be consumed. **Type**: `string` **Default**: `/get/stream` ### [](#timeout)`timeout` The maximum time to wait before a blocking, inactive connection is dropped (only applies to the `path` endpoint). **Type**: `string` **Default**: `5s` ### [](#ws_path)`ws_path` The path from which websocket connections can be established. **Type**: `string` **Default**: `/get/ws` --- # Page 158: iceberg **URL**: https://docs.redpanda.com/connect/components/outputs/iceberg.md --- # iceberg --- title: iceberg latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/iceberg page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/iceberg.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/iceberg.adoc description: Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API. page-git-created-date: "2026-03-05" page-git-modified-date: "2026-10-14" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API. Introduced in version 4.80.0. This output is well suited for migrating fanout pipelines from Kafka Connect to Redpanda Connect, and supports: - Multiple storage backends (S3, GCS, Azure) - Automatic table creation with schema detection - Partition transforms (year, month, day, hour, bucket, truncate) - Schema evolution (automatic column addition) - Transaction retry logic for concurrent writes ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). For tested configurations that tune this output for high throughput, covering the trade-off between preserving message ordering and maximizing throughput as well as keyed change-data-capture workloads that use `row_operation` with `upsert` or `delete`, see [Tune the Iceberg output for throughput](https://docs.redpanda.com/connect/guides/iceberg-output-tuning/). ### Common ```yml output: label: "" iceberg: catalog: url: "" # No default (required) warehouse: "" # No default (optional) auth: oauth2: server_uri: /v1/oauth/tokens client_id: "" # No default (required) client_secret: "" # No default (required) scope: "" # No default (optional) bearer: "" # No default (optional) aws_sigv4: {} namespace: "" # No default (required) table: "" # No default (required) storage: aws_s3: bucket: "" # No default (required) region: "" # No default (optional) endpoint: "" # No default (optional) gcp_cloud_storage: bucket: "" # No default (required) credentials_type: "" # No default (optional) credentials_file: "" # No default (optional) credentials_json: "" # No default (optional) azure_blob_storage: storage_account: "" # No default (required) container: "" # No default (required) storage_sas_token: "" # No default (optional) storage_connection_string: "" # No default (optional) storage_access_key: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" max_in_flight: 4 ``` ### Advanced ```yml output: label: "" iceberg: catalog: url: "" # No default (required) warehouse: "" # No default (optional) auth: oauth2: server_uri: /v1/oauth/tokens client_id: "" # No default (required) client_secret: "" # No default (required) scope: "" # No default (optional) bearer: "" # No default (optional) aws_sigv4: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) service: "" # No default (optional) headers: {} # No default (optional) tls_skip_verify: false namespace: "" # No default (required) table: "" # No default (required) case_sensitive_columns: true row_operation: insert identifier_fields: [] merge_strategy: merge-on-read storage: aws_s3: bucket: "" # No default (required) region: "" # No default (optional) endpoint: "" # No default (optional) force_path_style_urls: false credentials: id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) gcp_cloud_storage: bucket: "" # No default (required) endpoint: "" # No default (optional) credentials_type: "" # No default (optional) credentials_file: "" # No default (optional) credentials_json: "" # No default (optional) azure_blob_storage: storage_account: "" # No default (required) container: "" # No default (required) endpoint: "" # No default (optional) storage_sas_token: "" # No default (optional) storage_connection_string: "" # No default (optional) storage_access_key: "" # No default (optional) schema_evolution: enabled: false partition_spec: () table_location: "" # No default (optional) schema_metadata: "" new_column_type_mapping: "" # No default (optional) require_schema_metadata: false commit: manifest_merge_enabled: true max_snapshot_age: 24h max_retries: 3 cleanup_on_failure: true parquet: string_encoding: delta_length_byte_array compression: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` ## [](#catalog-integration)Catalog integration This output works with REST catalog implementations including Apache Polaris, AWS Glue Data Catalog, and Databricks Unity Catalog. ### [](#apache-polaris)Apache Polaris To use with [Apache Polaris](https://polaris.apache.org): - Set `catalog.url` to the Polaris REST endpoint (for example, `[http://localhost:8181/api/catalog](http://localhost:8181/api/catalog)`). - Set `catalog.warehouse` to the catalog name configured in Polaris. - Configure `catalog.auth.oauth2` with client credentials granted access to the catalog. ### [](#aws-glue-data-catalog)AWS Glue Data Catalog To use with AWS Glue Data Catalog: - Set `catalog.url` to `[https://glue..amazonaws.com/iceberg](https://glue.\.amazonaws.com/iceberg)` (the REST client appends the API version automatically). - Set `catalog.warehouse` to your AWS account ID (the Glue catalog identifier). - Set `schema_evolution.table_location` to an S3 prefix (for example, `s3://my-bucket/`) since Glue does not automatically assign table locations. - Configure `catalog.auth.aws_sigv4` with the appropriate region and set `service` to `glue`. - Configure `storage.aws_s3` with the same bucket and region. ### [](#azure-blob-storage-adls-gen2)Azure Blob Storage (ADLS Gen2) To use with Azure Data Lake Storage Gen2: - Configure `storage.azure_blob_storage` with your storage account name and container. - Authenticate using one of: `storage_access_key` (shared key), `storage_sas_token`, or `storage_connection_string`. - The storage account must have hierarchical namespace (HNS) enabled for ADLS Gen2 compatibility. ## [](#type-mapping)Type mapping | Bloblang type | Iceberg type | | --- | --- | | string | string | | bytes | binary | | bool | boolean | | number | double | | timestamp | timestamp (with timezone) | | object | struct | | array | list | ## [](#row-level-operations)Row-level operations By default, this output is append-only: every message becomes a new row (`row_operation: insert`), and existing configurations are unaffected. Set `row_operation` to apply a per-message operation, with `identifier_fields` defining the row identity: - `insert`: Append the row. This is an unconditional append, and is not keyed or deduplicated. - `upsert`: Replace any existing rows matching `identifier_fields`, then append this row. This is equivalent to Iceberg’s Flink UPSERT mode. - `delete`: Remove rows matching `identifier_fields`. `row_operation` supports interpolation, so the operation can be driven by the data itself, for example by mapping a change-data-capture stream’s operation field. No CDC-specific format is assumed. The field is named `row_operation` to distinguish it from Iceberg’s snapshot-level operation. `upsert` and `delete` require `identifier_fields` and use Iceberg merge-on-read equality deletes, which require table format version 2. A version-1 table is automatically upgraded to version 2 on the first `upsert` or `delete`. This upgrade is irreversible. ### [](#identifier-fields)Identifier fields `identifier_fields` must reference existing table columns of a primitive, non-floating-point type. A static `upsert` or `delete` is validated at startup. An interpolated `row_operation` is validated for each message at write time, so an empty `identifier_fields` is not caught until the first `upsert` or `delete` message arrives. Identifier columns of a temporal type (`timestamp`, `timestamptz`, `date`, `time`) must arrive as time values, not bare numbers. A numeric epoch is ambiguous as a delete key and is rejected at write time, so convert it to a timestamp upstream. If the table is partitioned, every partition source column must be one of the `identifier_fields`, because equality deletes are partition-scoped. When this output auto-creates a table through `schema_evolution`, the `identifier_fields` columns are created as required and registered as the table’s Iceberg identifier-field-ids, so downstream engines and other writers see the primary key. As a result, a null or missing value in an identifier column is rejected on write, even for `insert`. Identifier columns must therefore be present at creation, either in the first message or declared through `schema_metadata`. Pre-existing tables are never modified. ### [](#batching-and-ordering)Batching and ordering Within a single batch, the last `upsert` or `delete` for each `identifier_fields` key wins. Each batch containing an `upsert` or `delete` is committed as its own snapshot. These commits are never coalesced, which is required for correctness, so a high-throughput mutation workload produces one snapshot per batch. Size batches accordingly, and run regular table maintenance (snapshot expiry and compaction) to keep metadata manageable. Pure `insert`\-only batches keep the original append fast path, which does coalesce commits. Ordering only holds within a batch. With more than one batch in flight, concurrent batches can commit out of order, so a stale `upsert` might overwrite a newer one for the same key. Set `max_in_flight: 1` for keyed (change-data-capture) workloads to preserve per-key order. This is enforced by config linting whenever `row_operation` is anything other than a static `insert`. > ⚠️ **CAUTION** > > `insert` is an unconditional append, and is not keyed or deduplicated. For keyed data, including change-data-capture, map create and read events to `upsert`, never `insert`. Mixing `insert` with `upsert` or `delete` on the same key in one batch produces duplicate rows. ### [](#merge-strategies)Merge strategies `merge_strategy` controls how `upsert` and `delete` mutations are written to disk, which determines both write cost and which query engines can read the table. - `merge-on-read` (the default) writes Iceberg v2 equality-delete files and applies them at read time. Writes stay cheap and streaming-friendly, but only catalog-native or Flink-world engines, such as Apache Polaris, Flink, Trino, and Spark, can read equality deletes. Engine-backed catalogs such as Snowflake and the Databricks Unity Catalog cannot. It requires table format version 2, so a version-1 table is automatically upgraded to version 2 on the first `upsert` or `delete`. This upgrade is irreversible. - `copy-on-write` rewrites whole data files instead, so the table only ever holds plain data files, with no delete files. Every engine that reads Iceberg data files can read the result, including Snowflake, the Databricks Unity Catalog, Polaris, Flink, and Trino. Because it only ever writes plain data files, it works on version-1 or version-2 tables without forcing the version upgrade. Choose `merge-on-read` for streaming or high-throughput mutation into a lake read by Polaris, Flink, Trino, or Spark, where write cost must stay low. Choose `copy-on-write` when the table must be readable by Snowflake, the Databricks Unity Catalog, or another engine that cannot read equality deletes, and the workload is batch or moderate-throughput so the write amplification is acceptable. `copy-on-write` rewrites every data file that contains a touched key, so sort the table by the identifier key and use large batches to keep the number of rewritten files as low as possible. It also materializes the whole new-row batch in memory while the batch commits, so size keyed batches to stay within your process memory budget. If a commit fails, `commit.cleanup_on_failure` (enabled by default) removes the files that commit had already written, to limit orphaned objects in storage. Disable it only if you’d rather leave those files for Iceberg’s own orphan-file maintenance, such as snapshot expiry and `remove_orphan_files`, to reclaim. ### [](#timestamp-encoding)Timestamp encoding on existing tables Older versions of this output annotated no-timezone `timestamp` columns in the parquet files they wrote with `isAdjustedToUTC=true`: the annotation the Iceberg spec reserves for `timestamptz`. The stored microsecond instants are correct, and appends and most readers are unaffected, but the annotation makes some readers treat the column as UTC-adjusted, and it prevents `copy-on-write` from rewriting those files (the file’s annotation reads back as `timestamptz`, which cannot be written into a `timestamp` column). Current versions write the spec-correct `isAdjustedToUTC=false`. To guarantee an existing table never ends up with a mix of the two annotations, the encoding is pinned **per table** via the table property `redpanda-connect.timestamp-encoding` (`spec` or `legacy`): - Tables created by this output carry `redpanda-connect.timestamp-encoding: spec` from creation. - For an existing table without the property, the output resolves the encoding automatically on first contact and stamps the result onto the table: if the schema has no no-timezone `timestamp` column, or the table has no data files, it resolves `spec`; otherwise the output inspects one data file’s parquet footer and adopts whatever that file already contains (`legacy` for `isAdjustedToUTC=true`). A table that cannot be probed fails the write rather than risk mixing annotations, whether because a footer is unreadable, or because the first 1000 parquet data files probed (in manifest order) carry no no-timezone `timestamp` column at all (possible when the column was added by later schema evolution on a large table); in the latter case the error names the fix: set the property on the table explicitly and the probe never runs. The stamp itself is a `SetProperties` catalog commit, so the principal this output authenticates as needs permission to write table properties in addition to writing data; a principal without it turns a previously-healthy insert pipeline into a hard failure on first contact after upgrading, until the permission is granted or the property is set out-of-band. On a catalog that forbids **all** custom table properties, the set-it-explicitly escape hatch is unavailable by the same rule (the `redpanda-connect.` prefix carries connector semantics and is never silently stripped), so such writes fail loudly rather than risk mixing annotations. - Once stamped, the property is authoritative and the probe never runs again. An unrecognised property value is a hard error. A table pinned `legacy` keeps receiving the legacy annotation on every new file, byte-identical to what previous releases wrote, so appends and `merge-on-read` continue working unchanged forever. The one restriction is mutating `copy-on-write` (`upsert`/`delete`): it must rewrite existing files, which the legacy annotation prevents, so such writes fail upfront with an actionable error (pure `insert` batches still work). To migrate a legacy table to the spec encoding: rewrite/compact the table’s data files with an engine that writes the spec annotation (for example, Spark’s `rewrite_data_files`), then set the table property `redpanda-connect.timestamp-encoding` to `spec`, keeping any running instances of this output that write to the table stopped (or restarting them) around the migration: a live writer only re-reads the property when its writer is recreated. A table whose existing files already mix both annotations (for example one written to by several engines or connector versions over time) can be pinned either way by the probe, depending on which file it happens to read first, and a `copy-on-write` mutation on such a table may then fail mid-rewrite with the underlying library’s type-promotion error rather than the upfront migration message: compact or rewrite such a table to a single encoding before mutating it. Alternatively, keep the table on `merge-on-read`. ## [](#data-file-compression)Data file compression Compression of the parquet data files this output writes is resolved per table, in this order: 1. `parquet.compression`, if you set it. 2. otherwise the table’s own `write.parquet.compression-codec` property, if the table has one. 3. otherwise uncompressed. `parquet.compression` is optional rather than defaulted precisely so that step 2 is reachable: an unset field means "whatever the table says", which is not the same as explicitly choosing `uncompressed`. **Prefer the table property for compression settings whenever possible.** `parquet.compression` only applies to files this connector writes directly. The Iceberg library can perform additional writes, such as copy-on-write rewrites and equality-delete files, and these do not honour this field. They follow the table’s `write.parquet.compression-codec` property instead, which defaults to `zstd`. Setting the table property is the only way to get one codec for every file in the table. Use this field when you cannot set table properties, for example on Databricks Unity Catalog. > 📝 **NOTE** > > The uncompressed default applies only to a table whose property is genuinely absent. A table created through the Iceberg library (which includes tables this output creates itself) comes back carrying `write.parquet.compression-codec: zstd`, set at creation, so an otherwise unconfigured pipeline writing to such a table produces `zstd` throughout rather than anything uncompressed. Either way a table may end up holding a mixture of codecs, which is legal and transparent: parquet records its codec per column chunk, readers handle mixed files, and changing compression never requires rewriting existing data. **Codec support.** `snappy`, `gzip` and `zstd` are read by every engine this output targets. Parquet permits others that are not offered here: the original `lz4` codec was ambiguously specified and readers disagree on what it means, `lz4_raw` is its unambiguous replacement but has younger and less universal reader support, and `brotli` and `lzo` are patchily supported. If the table property names one of those it is reported in the log and the files this output writes are uncompressed instead. That governs only those files; the Iceberg library does map those codecs, so if the intent is that nothing writes one, change the table property rather than relying on this. **Cost.** Negligible impact on write throughput; the size savings depend heavily on how compressible your data is. ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#case_sensitive_columns)`case_sensitive_columns` Controls how message field names are matched against table column names, and how column references in the partition spec are resolved. When `true` (the default), names must match exactly. When `false`, matching is case-insensitive; set this when your downstream catalog or query engine treats column names as case-insensitive (the iceberg specification’s recommended convention) so that, for example, a message keyed `"COLUMN"` lands in an existing `column` rather than triggering schema evolution. Ambiguous case-only duplicates in the input are rejected. Requires version 4.89.3 or later. **Type**: `bool` **Default**: `true` ### [](#catalog)`catalog` REST catalog configuration. **Type**: `object` ### [](#catalog-auth)`catalog.auth` Authentication configuration for the REST catalog. Only one authentication method can be active at a time. **Type**: `object` ### [](#catalog-auth-aws_sigv4)`catalog.auth.aws_sigv4` AWS SigV4 authentication (for AWS Glue Data Catalog or API Gateway). **Type**: `object` ### [](#catalog-auth-aws_sigv4-credentials)`catalog.auth.aws_sigv4.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). Requires version 4.84.0 or later. **Type**: `object` ### [](#catalog-auth-aws_sigv4-credentials-from_ec2_role)`catalog.auth.aws_sigv4.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#catalog-auth-aws_sigv4-credentials-id)`catalog.auth.aws_sigv4.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-profile)`catalog.auth.aws_sigv4.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-role)`catalog.auth.aws_sigv4.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-role_external_id)`catalog.auth.aws_sigv4.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-secret)`catalog.auth.aws_sigv4.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-token)`catalog.auth.aws_sigv4.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#catalog-auth-aws_sigv4-endpoint)`catalog.auth.aws_sigv4.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. Requires version 4.84.0 or later. **Type**: `string` ### [](#catalog-auth-aws_sigv4-region)`catalog.auth.aws_sigv4.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#catalog-auth-aws_sigv4-service)`catalog.auth.aws_sigv4.service` AWS service name for SigV4 signing. **Type**: `string` ### [](#catalog-auth-aws_sigv4-tcp)`catalog.auth.aws_sigv4.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.84.0 or later. **Type**: `object` ### [](#catalog-auth-aws_sigv4-tcp-connect_timeout)`catalog.auth.aws_sigv4.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive)`catalog.auth.aws_sigv4.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-count)`catalog.auth.aws_sigv4.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-idle)`catalog.auth.aws_sigv4.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-interval)`catalog.auth.aws_sigv4.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#catalog-auth-aws_sigv4-tcp-tcp_user_timeout)`catalog.auth.aws_sigv4.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#catalog-auth-bearer)`catalog.auth.bearer` Static bearer token for authentication. For testing only, not recommended for production. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#catalog-auth-oauth2)`catalog.auth.oauth2` OAuth2 authentication configuration. **Type**: `object` ### [](#catalog-auth-oauth2-client_id)`catalog.auth.oauth2.client_id` OAuth2 client identifier. **Type**: `string` ### [](#catalog-auth-oauth2-client_secret)`catalog.auth.oauth2.client_secret` OAuth2 client secret. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#catalog-auth-oauth2-scope)`catalog.auth.oauth2.scope` OAuth2 scope to request. **Type**: `string` ### [](#catalog-auth-oauth2-server_uri)`catalog.auth.oauth2.server_uri` OAuth2 token endpoint URI. **Type**: `string` **Default**: `/v1/oauth/tokens` ### [](#catalog-headers)`catalog.headers` Custom HTTP headers to include in all requests to the catalog. **Type**: `object` ```yaml # Examples: headers: X-Api-Key: your-api-key ``` ### [](#catalog-tls_skip_verify)`catalog.tls_skip_verify` Skip TLS certificate verification. Not recommended for production. **Type**: `bool` **Default**: `false` ### [](#catalog-url)`catalog.url` The REST catalog endpoint URL. **Type**: `string` ```yaml # Examples: url: http://localhost:8181/api/catalog # --- url: https://polaris.example.com/api/catalog # --- url: https://glue.us-east-1.amazonaws.com/iceberg ``` ### [](#catalog-warehouse)`catalog.warehouse` The REST catalog warehouse. **Type**: `string` ```yaml # Examples: warehouse: redpanda-catalog ``` ### [](#commit)`commit` Commit behavior configuration. **Type**: `object` ### [](#commit-cleanup_on_failure)`commit.cleanup_on_failure` Whether to remove the files a failed commit had already written. A commit’s data files (and, under `merge-on-read`, its equality-delete files) are written to storage before the catalog commit, so a failed commit leaves them referenced by no snapshot; the default `true` removes them best-effort to limit orphaned objects. The same sweep reclaims the superseded files of earlier attempts when a `copy-on-write` commit succeeds on retry (each attempt re-stages fresh rewrites). This applies to every write path: append, `merge-on-read`, and `copy-on-write`. Cleanup is already skipped automatically whenever a commit’s outcome is ambiguous, since such a commit may still land server-side, and deleting files a landed snapshot references would corrupt the table. Those leftovers are always deferred to table maintenance regardless of this setting. Setting this to `false` disables cleanup entirely, as a safety valve: writes then never delete anything, and every failed commit leaves its files orphaned in storage for Iceberg orphan-file maintenance (snapshot expiry plus `remove_orphan_files`) to reclaim. See the [Merge strategies](#merge-strategies) section above for that maintenance guidance. Requires version 4.105.0 or later. **Type**: `bool` **Default**: `true` ### [](#commit-manifest_merge_enabled)`commit.manifest_merge_enabled` Merge small manifest files during commits to reduce metadata overhead. **Type**: `bool` **Default**: `true` ### [](#commit-max_retries)`commit.max_retries` Maximum number of times to retry a failed transaction commit. **Type**: `int` **Default**: `3` ### [](#commit-max_snapshot_age)`commit.max_snapshot_age` Maximum age of snapshots to retain for time-travel queries. Set to zero to disable removing old snapshots. **Type**: `string` **Default**: `24h` ### [](#identifier_fields)`identifier_fields[]` The columns forming the row identity (the Iceberg identifier fields, or equality-delete key) used by `upsert` and `delete`. Required when `row_operation` can evaluate to `upsert` or `delete`, and must reference existing table columns of a primitive, non-floating-point type. See the Row-level operations section above for the full constraints, including the temporal-type and partitioning rules and when the requirement is enforced. Requires version 4.99.0 or later. **Type**: `array` **Default**: `[]` ```yaml # Examples: identifier_fields: - id # --- identifier_fields: - tenant_id - user_id ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `4` ### [](#merge_strategy)`merge_strategy` How `upsert` and `delete` are materialised on disk. - `merge-on-read` (the default) writes Iceberg v2 equality-delete files. Deletes are applied at read time, so writes stay cheap and streaming-friendly, but only catalog-native / Flink-world engines can read the result. Engine-backed catalogs such as Snowflake and the Databricks Unity Catalog cannot read equality deletes. - `copy-on-write` rewrites whole data files so the table only ever contains plain data files (no delete files), which every engine can read, including Snowflake and Databricks Unity Catalog. It works on version-1 or version-2 tables and never forces the irreversible v1→v2 upgrade. The trade-off is heavy write amplification: each mutating batch rewrites every data file that contains a touched key, so it is a batch / moderate-throughput mode. Sort the table by the identifier key and use large batches so each rewrite touches as few files as possible. See the [Merge strategies](#merge-strategies) section above for the full decision guide, copy-on-write support matrix (column and merge-key types, partitioning, table format), and maintenance guidance. Requires version 4.105.0 or later. **Type**: `string` **Default**: `merge-on-read` **Options**: `merge-on-read`, `copy-on-write` ### [](#namespace)`namespace` The Iceberg namespace for the table, dot delimiters are split as nested namespaces. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: namespace: analytics.events # --- namespace: production ``` ### [](#parquet)`parquet` Parquet writer configuration. Requires version 4.90.3 or later. **Type**: `object` ### [](#parquet-compression)`parquet.compression` The compression codec for data files this output writes. **Optional on purpose**: when it is not set, the codec is taken from the table’s own `write.parquet.compression-codec` property, and when that is absent too, data files are written uncompressed. Setting the table property rather than this field is usually the better choice, because the property is also honoured by the copy-on-write rewrite path (which writes its files inside the Iceberg library, out of reach of this field). The property is what gets every file in the table onto one codec, whereas this field governs only the data files this output writes itself: appends and merge-on-read data files, not copy-on-write rewrites or equality-delete files. Use this field when the property cannot be set, for example on catalogs that reject client-set table properties. If you rely on the table property instead, use a lower-case codec name: this field accepts any casing, but the Iceberg library’s own property lookup is lower-case only and silently falls back to uncompressed for anything else. A property of `ZSTD` would give you compressed appends and uncompressed copy-on-write rewrites. Only codecs that every engine this output targets can read are offered. If the table property names something else (`lz4`, `lz4_raw`, `brotli`, `lzo`), it is reported in the log and the data files this output writes are uncompressed instead. Measurement found no throughput penalty worth planning around, at one core or four; the size benefit though is entirely data-dependent, ranging from ~15x smaller on a repetitive record shape to ~2% on random content. Note also that the uncompressed fallback applies only when the table property is absent, and tables created through the Iceberg library carry a `zstd` property by default. See [Data file compression](#data-file-compression) for the resolution order and the copy-on-write caveat. Requires version 4.109.0 or later. **Type**: `string` **Options**: `uncompressed`, `snappy`, `gzip`, `zstd` ### [](#parquet-string_encoding)`parquet.string_encoding` The encoding to use for string and binary columns. Use `plain` for compatibility with readers that do not support `DELTA_LENGTH_BYTE_ARRAY` encoding, such as AWS Redshift Spectrum. **Type**: `string` **Default**: `delta_length_byte_array` **Options**: `plain`, `delta_length_byte_array` ### [](#row_operation)`row_operation` The row-level operation to apply for each message: `insert` (append), `upsert` (replace rows matching `identifier_fields`, then append), or `delete` (remove rows matching `identifier_fields`). Supports interpolation so the operation can be driven by the data, such as a change-data-capture stream’s operation field. Defaults to `insert`, preserving the original append-only behavior. See the Row-level operations section above for the full semantics, the format-version-2 upgrade, batching behavior, and important caveats. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.99.0 or later. **Type**: `string` **Default**: `insert` ```yaml # Examples: row_operation: insert # --- row_operation: ${! metadata("op") } # --- row_operation: '${! this.op == "d" ? "delete" : "upsert" }' ``` ### [](#schema_evolution)`schema_evolution` Schema evolution configuration. **Type**: `object` ### [](#schema_evolution-enabled)`schema_evolution.enabled` Enable automatic schema evolution. When enabled, new columns will be automatically added to the table. **Type**: `bool` **Default**: `false` ### [](#schema_evolution-new_column_type_mapping)`schema_evolution.new_column_type_mapping` An optional Bloblang mapping to customize column types during schema evolution. This mapping is executed for each new column and can override the inferred or schema-metadata-derived type. The mapping receives an object with fields `name` (column name), `path` (dot-separated path), `value` (sample value), `inferred_type` (the type that would be used without this mapping), `message` (the full message body), `namespace`, and `table`. It must return a string with a valid Iceberg type name: `boolean`, `int`, `long`, `float`, `double`, `string`, `binary`, `date`, `time`, `timestamp`, `timestamptz`, `uuid`, `decimal(p,s)`, or `fixed[n]`. Requires version 4.85.0 or later. **Type**: `string` ### [](#schema_evolution-partition_spec)`schema_evolution.partition_spec` A Bloblang expression to evaluate when a new table is created to determine the table’s partition spec. The result of the mapping should be an Iceberg partition spec in the same string format as the [Redpanda Streaming topic property](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/#use-custom-partitioning). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `()` ```yaml # Examples: partition_spec: (col1) # --- partition_spec: (nested.col) # --- partition_spec: (year(my_ts_col)) # --- partition_spec: (year(my_ts_col), col2) # --- partition_spec: (hour(my_ts_col), truncate(42, col2)) # --- partition_spec: (day(my_ts_col), bucket(4, nested.col)) # --- partition_spec: (day(my_ts_col), void(`non.nested column.with.dots`), identity(nested.column)) ``` ### [](#schema_evolution-require_schema_metadata)`schema_evolution.require_schema_metadata` When `true`, writing a numeric value into a `timestamp`, `timestamptz`, `date`, or `time` column without `schema_metadata` registered for that column is a hard error. The default `false` permits a fallback path that interprets bare numeric timestamps as Unix seconds and bare numeric times as already-microseconds; convenient, but silently wrong if upstream produced milliseconds. Enable this when you cannot guarantee the upstream attaches schema metadata and want to fail loudly rather than corrupt dates by ~50,000 years. No effect on time-typed columns receiving `time.Time`/`time.Duration` Go values, which carry their own unit unambiguously, and no effect on non-time columns. Requires `schema_metadata` to be set. Requires version 4.94.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_evolution-schema_metadata)`schema_evolution.schema_metadata` The name of a message metadata field containing a schema definition. When set, the schema is used to determine column types during schema evolution and table creation instead of inferring types from values. The schema must be in the standard common schema format (the same format used by the `parquet_encode` processor’s `schema_metadata` field). For batches of messages, the first message’s schema is used. Record presence drives schema shape: fields declared in the schema metadata that are absent from the record are not added to the table, while the metadata controls column ordering, naming, and types for fields that are present. In case-insensitive mode, top-level column names use the metadata’s casing; record keys are matched by case-folding and the metadata’s name is what lands in the table. Requires version 4.85.0 or later. **Type**: `string` **Default**: `""` ### [](#schema_evolution-table_location)`schema_evolution.table_location` A prefix used as the location for new tables when the catalog does not automatically assign one. For example, AWS Glue requires explicit table locations. When set, table locations are derived by appending the namespace and table name to the prefix, in the form `/`. **Type**: `string` ```yaml # Examples: table_location: s3://my-iceberg-bucket/ ``` ### [](#storage)`storage` Storage backend configuration for data files. Exactly one of `aws_s3`, `gcp_cloud_storage`, or `azure_blob_storage` must be specified. **Type**: `object` ### [](#storage-aws_s3)`storage.aws_s3` S3 storage configuration. **Type**: `object` ### [](#storage-aws_s3-bucket)`storage.aws_s3.bucket` The S3 bucket name. **Type**: `string` ```yaml # Examples: bucket: my-iceberg-data ``` ### [](#storage-aws_s3-credentials)`storage.aws_s3.credentials` Static AWS credentials for S3 access. When not specified, credentials are loaded from the default AWS credential chain. **Type**: `object` ### [](#storage-aws_s3-credentials-id)`storage.aws_s3.credentials.id` The AWS access key ID. **Type**: `string` ### [](#storage-aws_s3-credentials-secret)`storage.aws_s3.credentials.secret` The AWS secret access key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#storage-aws_s3-credentials-token)`storage.aws_s3.credentials.token` The AWS session token, required when using short term credentials. **Type**: `string` ### [](#storage-aws_s3-endpoint)`storage.aws_s3.endpoint` Custom endpoint for S3-compatible storage (for example, MinIO). **Type**: `string` ```yaml # Examples: endpoint: http://localhost:9000 ``` ### [](#storage-aws_s3-force_path_style_urls)`storage.aws_s3.force_path_style_urls` Forces the client API to use path style URLs, which is often required when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#storage-aws_s3-region)`storage.aws_s3.region` The AWS region. **Type**: `string` ```yaml # Examples: region: us-west-2 ``` ### [](#storage-azure_blob_storage)`storage.azure_blob_storage` Azure Blob Storage (ADLS Gen2) configuration. **Type**: `object` ### [](#storage-azure_blob_storage-container)`storage.azure_blob_storage.container` The Azure blob container name. **Type**: `string` ```yaml # Examples: container: iceberg-data ``` ### [](#storage-azure_blob_storage-endpoint)`storage.azure_blob_storage.endpoint` Custom endpoint for Azure-compatible storage. **Type**: `string` ### [](#storage-azure_blob_storage-storage_access_key)`storage.azure_blob_storage.storage_access_key` Azure storage access key for shared key authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#storage-azure_blob_storage-storage_account)`storage.azure_blob_storage.storage_account` The Azure storage account name. **Type**: `string` ```yaml # Examples: storage_account: mystorageaccount ``` ### [](#storage-azure_blob_storage-storage_connection_string)`storage.azure_blob_storage.storage_connection_string` Azure storage connection string. Use this or other auth methods, not both. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#storage-azure_blob_storage-storage_sas_token)`storage.azure_blob_storage.storage_sas_token` SAS token for authentication. Prefix with the container name followed by a dot if container-specific. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#storage-gcp_cloud_storage)`storage.gcp_cloud_storage` Google Cloud Storage configuration. **Type**: `object` ### [](#storage-gcp_cloud_storage-bucket)`storage.gcp_cloud_storage.bucket` The GCS bucket name. **Type**: `string` ```yaml # Examples: bucket: my-iceberg-data ``` ### [](#storage-gcp_cloud_storage-credentials_file)`storage.gcp_cloud_storage.credentials_file` Path to a GCP credentials JSON file. **Type**: `string` ### [](#storage-gcp_cloud_storage-credentials_json)`storage.gcp_cloud_storage.credentials_json` GCP credentials JSON content. Use this or `credentials_file`, not both. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#storage-gcp_cloud_storage-credentials_type)`storage.gcp_cloud_storage.credentials_type` The type of credentials to use. Valid values: `service_account`, `authorized_user`, `impersonated_service_account`, `external_account`. **Type**: `string` ```yaml # Examples: credentials_type: service_account ``` ### [](#storage-gcp_cloud_storage-endpoint)`storage.gcp_cloud_storage.endpoint` Custom endpoint for GCS-compatible storage. **Type**: `string` ### [](#table)`table` The destination Iceberg table name. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table: user_events # --- table: events_${!meta("topic")} ``` ## [](#examples)Examples ### [](#change-data-capture-upsertdelete)Change-data-capture upsert/delete Materialize a change-data-capture stream into an Iceberg table. A mapping derives the row operation from the source’s operation field (here Debezium’s `op`: ``c`reate / `r`ead / `u`pdate map to `upsert``, ``d`elete maps to `delete``) and selects the row image, while `identifier_fields` is the primary key. Note that creates map to `upsert`, never `insert`, so re-delivered or snapshot rows do not duplicate. ```yaml input: redpanda: seed_brokers: [ localhost:9092 ] topics: [ dbserver.inventory.customers ] consumer_group: iceberg_sink pipeline: processors: - mapping: | meta op = match this.op { "d" => "delete", _ => "upsert", } # Debezium puts the row image in 'after' (or 'before' for deletes). root = this.after | this.before output: iceberg: catalog: url: http://localhost:8181/api/catalog namespace: inventory table: customers row_operation: ${! metadata("op") } identifier_fields: [ id ] # Keyed writes must stay ordered: a single batch in flight prevents # concurrent batches from committing a stale update over a newer one. max_in_flight: 1 storage: aws_s3: bucket: my-iceberg-data region: us-east-1 ``` ### [](#cdc-into-a-snowflake-or-databricks-readable-table-copy-on-write)CDC into a Snowflake- or Databricks-readable table (copy-on-write) Materialize a change-data-capture stream into an Iceberg table that must be read by an engine-backed catalog such as Snowflake or the Databricks Unity Catalog, which cannot read merge-on-read equality deletes. `merge_strategy: copy-on-write` rewrites whole data files so the table only ever holds plain data files that every engine can read. As with any keyed workload it requires `max_in_flight: 1`; because copy-on-write rewrites every file containing a touched key, prefer large batches and a table sorted by the identifier key, and run regular compaction and snapshot expiry. ```yaml input: redpanda: seed_brokers: [ localhost:9092 ] topics: [ dbserver.inventory.customers ] consumer_group: iceberg_sink_cow pipeline: processors: - mapping: | meta op = match this.op { "d" => "delete", _ => "upsert", } root = this.after | this.before output: iceberg: catalog: url: http://localhost:8181/api/catalog namespace: inventory table: customers row_operation: ${! metadata("op") } identifier_fields: [ id ] # copy-on-write produces only plain data files, so Snowflake and the # Databricks Unity Catalog can read the result (unlike equality deletes). merge_strategy: copy-on-write # Keyed writes must stay ordered; a single batch in flight prevents a stale # update from overwriting a newer one for the same key. max_in_flight: 1 # Amortise the file rewrites over larger commits. batching: count: 5000 period: 30s storage: aws_s3: bucket: my-iceberg-data region: us-east-1 ``` --- # Page 159: inproc **URL**: https://docs.redpanda.com/connect/components/outputs/inproc.md --- # inproc --- title: inproc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/inproc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/inproc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/inproc.adoc description: Sends data directly to Redpanda Connect inputs in the same process by connecting to a unique ID, to link isolated streams when running in streams mode. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- ```yml output: label: "" inproc: "" ``` Sends data directly to Redpanda Connect inputs by connecting to a unique ID. This allows you to hook up isolated streams whilst running Redpanda Connect in [streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/), it is NOT recommended that you connect the inputs of a stream with an output of the same stream, as feedback loops can lead to deadlocks in your message flow. It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs. --- # Page 160: kafka_franz **URL**: https://docs.redpanda.com/connect/components/outputs/kafka_franz.md --- # kafka_franz --- title: kafka_franz latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/kafka_franz page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/kafka_franz.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/kafka_franz.adoc description: A Kafka output using the Franz Kafka client library. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) components. The `kafka_franz` output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgments back to the input. This output often outperforms the traditional `kafka` output, as well as providing more useful logs and error messages. This output uses the [Franz Kafka client library](https://github.com/twmb/franz-go). Introduced in version 3.61.0. #### Common ```yml output: label: "" kafka_franz: seed_brokers: [] # No default (required) topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" kafka_franz: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) partitioner: "" # No default (optional) idempotent_write: true acks: all compression: snappy allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: "0" max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s ``` ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. If set to `false`, the topic must already exist. Requires version 4.58.0 or later. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. Requires version 4.35.0 or later. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. Requires version 4.21.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` The compression codec for produced record batches. A batch that compression does not make smaller is sent uncompressed, whichever codec you choose. **Type**: `string` **Default**: `snappy` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.53.0 or later. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the `IDEMPOTENT_WRITE` permission on the `CLUSTER` resource. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally. Requires version 4.24.0 or later. **Type**: `bool` **Default**: `true` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. Requires version 4.92.0 or later. **Type**: `int` **Default**: `10000` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to send in parallel at any given time. **Type**: `int` **Default**: `10` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. While `idempotent_write` is enabled (the default) this must be `1`, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set `idempotent_write` to `false`, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s `max_in_flight` field, which counts message batches being written in parallel rather than produce requests on the wire. A value of `1` is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests. Requires version 4.92.0 or later. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum size of a produced record batch in bytes. A `MESSAGE_TOO_LARGE` error is returned if a batch exceeds this limit. This field maps to the `max.message.bytes` Kafka property. Ensure the Redpanda broker’s `kafka_batch_max_bytes` property is at least as large as this value, see [kafka\_batch\_max\_bytes](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_batch_max_bytes). **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata)`metadata` Determines which metadata values are added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#partition)`partition` An optional explicit partition to set for each message. This field is only relevant when the `partitioner` is set to `manual`. The provided interpolation string must be a valid integer. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.13.0 or later. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. Requires version 4.53.0 or later. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. Requires version 4.3.0 or later. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. Requires version 4.3.0 or later. **Type**: `string` **Default**: `10s` ### [](#timestamp_ms)`timestamp_ms` An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both `timestamp` and `timestamp_ms`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.40.0 or later. **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 161: kafka **URL**: https://docs.redpanda.com/connect/components/outputs/kafka.md --- # kafka --- title: kafka latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/kafka page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/kafka.adoc description: The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-12" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) components. The `kafka` output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input. #### Common ```yml output: label: "" kafka: addresses: [] # No default (required) topic: "" # No default (required) target_version: "" # No default (optional) key: "" partitioner: fnv1a_hash compression: none static_headers: {} # No default (optional) metadata: exclude_prefixes: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" kafka: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" access_token: "" token_cache: "" token_key: "" topic: "" # No default (required) client_id: benthos target_version: "" # No default (optional) rack_id: "" key: "" partitioner: fnv1a_hash partition: "" custom_topic_creation: enabled: false partitions: -1 replication_factor: -1 compression: none static_headers: {} # No default (optional) metadata: exclude_prefixes: [] inject_tracing_map: "" # No default (optional) max_in_flight: 64 idempotent_write: false ack_replicas: false max_msg_bytes: 1000000 timeout: 5s retry_as_batch: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_retries: 0 backoff: initial_interval: 3s max_interval: 10s max_elapsed_time: 30s timestamp_ms: "" # No default (optional) ``` The configuration field `ack_replicas` determines whether Redpanda Connect waits for acknowledgement from all replicas or just a single broker. Both the `key` and `topic` fields can be dynamically set using function interpolations described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). [Metadata](https://docs.redpanda.com/connect/configuration/metadata/) will be added to each message sent as headers (version 0.11+), but can be restricted using the field [`metadata`](#metadata). ## [](#strict-ordering-and-retries)Strict ordering and retries When strict ordering is required for messages written to topic partitions it is important to ensure that both the field `max_in_flight` is set to `1` and that the field `retry_as_batch` is set to `true`. You must also ensure that failed batches are never rerouted back to the same output. This can be done by setting the field `max_retries` to `0` and `backoff.max_elapsed_time` to empty, which will apply back pressure indefinitely until the batch is sent successfully. However, this also means that manual intervention will eventually be required in cases where the batch cannot be sent due to configuration problems such as an incorrect `max_msg_bytes` estimate. A less strict but automated alternative would be to route failed batches to a dead letter queue using a [`fallback` broker](https://docs.redpanda.com/connect/components/outputs/fallback/), but this would allow subsequent batches to be delivered in the meantime whilst those failed batches are dealt with. ## [](#troubleshooting)Troubleshooting This component is deprecated. Use the [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) to write to Kafka brokers. - I’m seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tls-enabled) if applicable. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#ack_replicas)`ack_replicas` Ensure that messages have been copied across all replicas before acknowledging receipt. **Type**: `bool` **Default**: `false` ### [](#addresses)`addresses[]` A list of broker addresses to connect to. List items that contain commas are expanded into multiple addresses. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9092" # --- addresses: - "localhost:9041,localhost:9042" # --- addresses: - "localhost:9041" - "localhost:9042" ``` ### [](#backoff)`backoff` The exponential backoff between attempts to resend messages that failed to send. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `3s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. Setting this value to a zeroed duration (such as `0s`) will result in unbounded retries. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `10s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_id)`client_id` The client ID sent to brokers with every request, which lets brokers attribute requests to this client in their logs, metrics, and client quotas. **Type**: `string` **Default**: `benthos` ### [](#compression)`compression` The compression algorithm to use. **Type**: `string` **Default**: `none` **Options**: `none`, `snappy`, `lz4`, `gzip`, `zstd` ### [](#custom_topic_creation)`custom_topic_creation` If enabled, topics will be created with the specified number of partitions and replication factor if they do not already exist. Requires version 4.23.0 or later. **Type**: `object` ### [](#custom_topic_creation-enabled)`custom_topic_creation.enabled` Whether to enable custom topic creation. **Type**: `bool` **Default**: `false` ### [](#custom_topic_creation-partitions)`custom_topic_creation.partitions` The number of partitions to create for new topics. Leave at -1 to use the broker configured default. Must be >= 1. **Type**: `int` **Default**: `-1` ### [](#custom_topic_creation-replication_factor)`custom_topic_creation.replication_factor` The replication factor to use for new topics. Leave at -1 to use the broker configured default. Must be an odd number, and less then or equal to the number of brokers. **Type**: `int` **Default**: `-1` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER` and can be disabled if this permission is not available. Requires version 4.26.0 or later. **Type**: `bool` **Default**: `false` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 3.45.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_msg_bytes)`max_msg_bytes` The maximum size in bytes of messages sent to the target topic. **Type**: `int` **Default**: `1000000` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#metadata)`metadata` Specify which metadata values are added to each record as Kafka headers. Headers are only sent when `target_version` is `0.11.0` or later. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#partition)`partition` The manually-specified partition to publish messages to, relevant only when the field `partitioner` is set to `manual`. Must be able to parse as a 32-bit integer. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#partitioner)`partitioner` The partitioning algorithm to use. **Type**: `string` **Default**: `fnv1a_hash` **Options**: `fnv1a_hash`, `murmur2_hash`, `random`, `round_robin`, `manual` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#retry_as_batch)`retry_as_batch` When enabled forces an entire batch of messages to be retried if any individual message fails on a send, otherwise only the individual messages that failed are retried. Disabling this helps to reduce message duplicates during intermittent errors, but also makes it impossible to guarantee strict ordering of messages. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-access_token)`sasl.access_token` A static OAUTHBEARER access token **Type**: `string` **Default**: `""` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism, if left empty SASL authentication is not used. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS. | | SCRAM-SHA-256 | Authentication using the SCRAM-SHA-256 mechanism. | | SCRAM-SHA-512 | Authentication using the SCRAM-SHA-512 mechanism. | | none | Default, no SASL authentication. | ### [](#sasl-password)`sasl.password` A PLAIN password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-token_cache)`sasl.token_cache` Instead of using a static `access_token` allows you to query a [`cache`](https://docs.redpanda.com/connect/components/caches/about/) resource to fetch OAUTHBEARER tokens from **Type**: `string` **Default**: `""` ### [](#sasl-token_key)`sasl.token_key` Required when using a `token_cache`, the key to query the cache with for tokens. **Type**: `string` **Default**: `""` ### [](#sasl-user)`sasl.user` A PLAIN username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#static_headers)`static_headers` An optional map of static headers that should be added to messages in addition to metadata. **Type**: `object` ```yaml # Examples: static_headers: first-static-header: value-1 second-static-header: value-2 ``` ### [](#target_version)`target_version` The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version. **Type**: `string` ```yaml # Examples: target_version: 2.1.0 # --- target_version: 3.1.0 ``` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `5s` ### [](#timestamp_ms)`timestamp_ms` An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both `timestamp` and `timestamp_ms`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.40.0 or later. **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 162: mongodb **URL**: https://docs.redpanda.com/connect/components/outputs/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/mongodb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/mongodb.adoc description: Inserts items into a MongoDB collection. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Inserts items into a MongoDB collection. Introduced in version 3.43.0. #### Common ```yml output: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) operation: update-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos aws: enabled: false region: "" # No default (optional) session_duration: 1h id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) collection: "" # No default (required) operation: update-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight, in parallel, for improved performance. You can tune the maximum number of in flight messages (or message batches) using the `max_in_flight` field. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. Requires version 4.32.0 or later. **Type**: `string` **Default**: `benthos` ### [](#aws)`aws` AWS IAM authentication using the `MONGODB-AWS` mechanism, for example against MongoDB Atlas. When enabled, IAM credentials are used instead of a static username and password. Role-derived session credentials are resolved when the component connects and are re-resolved whenever it reconnects. The `mongodb` processor and cache establish their client once at creation and cannot refresh expiring session credentials, so `role`, `roles` and session tokens are rejected for those components; use the ambient credential chain or long-lived access keys with them. For long-running pipelines, prefer the ambient credential chain (leave keys and roles unset), which the driver refreshes automatically. Requires version 4.106.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication using the driver-native `MONGODB-AWS` mechanism. The MongoDB Atlas database user must be created with the AWS IAM authentication type, and connections require TLS. When no static credentials or roles are configured, the ambient AWS credential chain (environment variables, EC2 instance profile, EKS pod role) is used and expiring credentials are refreshed automatically. **Type**: `bool` **Default**: `false` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region used when assuming roles (for STS calls). Only used when `role` or `roles` are configured; the ambient and static-key paths ignore it. If no region is specified then the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Cannot be combined with `roles`; use the `roles` array instead when chaining multiple roles. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. Cannot be combined with `role`. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-session_duration)`aws.session_duration` The duration of the STS session requested when assuming roles. AWS requires at least 15 minutes and caps sessions created through role chaining at one hour. Only used when `role` or `roles` are configured. For long-running pipelines, prefer the ambient credential chain over a fixed session duration, since the driver refreshes ambient credentials automatically as they near expiry. **Type**: `string` **Default**: `1h` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#collection)`collection` The name of the target collection. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#document_map)`document_map` A Bloblang map that represents a document to store in MongoDB, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, `update-one`, and `aggregate`. **Type**: `string` **Default**: `""` ```yaml # Examples: document_map: |- root.a = this.foo root.b = this.bar ``` ### [](#filter_map)`filter_map` A Bloblang map that represents a filter for a MongoDB command, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `filter_map` parameter is required for all database operations except `insert-one`. The `filter_map` is used to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion. **Type**: `string` **Default**: `""` ```yaml # Examples: filter_map: |- root.a = this.foo root.b = this.bar ``` ### [](#hint_map)`hint_map` A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). This map is optional, and is used with all operations except `insert-one`. Define a `hint_map` to improve performance when finding documents in the MongoDB database. **Type**: `string` **Default**: `""` ```yaml # Examples: hint_map: |- root.a = this.foo root.b = this.bar ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` The MongoDB database operation to perform. **Type**: `string` **Default**: `update-one` **Options**: `insert-one`, `delete-one`, `delete-many`, `replace-one`, `update-one` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#upsert)`upsert` The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created. Requires version 3.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` ### [](#write_concern)`write_concern` The [write concern settings](https://www.mongodb.com/docs/manual/reference/write-concern/) for the MongoDB connection. **Type**: `object` ### [](#write_concern-j)`write_concern.j` The `j` option requests acknowledgement from MongoDB that write operations are written to the journal. **Type**: `bool` **Default**: `false` ### [](#write_concern-w)`write_concern.w` The `w` option requests acknowledgement that write operations propagate to the specified number of MongoDB instances. Set it to `majority` to wait for a calculated majority of nodes to acknowledge the write operation, to an integer to specify the minimum number of nodes that must acknowledge the operation, or to the name of a custom write concern configured in the cluster. **Type**: `string` **Default**: `majority` ### [](#write_concern-w_timeout)`write_concern.w_timeout` The write concern timeout. **Type**: `string` **Default**: `""` --- # Page 163: mqtt **URL**: https://docs.redpanda.com/connect/components/outputs/mqtt.md --- # mqtt --- title: mqtt latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/mqtt page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/mqtt.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/mqtt.adoc description: Pushes messages to an MQTT broker. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Pushes messages to an MQTT broker. Introduced in version 4.37.0. #### Common ```yml output: label: "" mqtt: urls: [] # No default (required) client_id: "" connect_timeout: 30s topic: "" # No default (required) qos: 1 write_timeout: 3s retained: false max_in_flight: 64 ``` #### Advanced ```yml output: label: "" mqtt: urls: [] # No default (required) client_id: "" dynamic_client_id_suffix: "" # No default (optional) connect_timeout: 30s will: enabled: false qos: 0 retained: false topic: "" payload: "" user: "" password: "" keepalive: 30 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topic: "" # No default (required) qos: 1 write_timeout: 3s retained: false retained_interpolated: "" # No default (optional) max_in_flight: 64 ``` The `topic` field can be dynamically set using function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#client_id)`client_id` The client identifier sent to the MQTT broker when connecting, which the broker uses to identify the client session. Use `dynamic_client_id_suffix` to give each run of the pipeline a unique identifier. **Type**: `string` **Default**: `""` ### [](#connect_timeout)`connect_timeout` The maximum amount of time to wait in order to establish a connection before the attempt is abandoned. Requires version 3.58.0 or later. **Type**: `string` **Default**: `30s` ```yaml # Examples: connect_timeout: 1s # --- connect_timeout: 500ms ``` ### [](#dynamic_client_id_suffix)`dynamic_client_id_suffix` Append a dynamically generated suffix to the specified `client_id` on each run of the pipeline. This can be useful when clustering Redpanda Connect producers. **Type**: `string` | Option | Summary | | --- | --- | | nanoid | append a nanoid of length 21 characters | ### [](#keepalive)`keepalive` Max seconds of inactivity before a keepalive message is sent. **Type**: `int` **Default**: `30` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` A password to connect with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#qos)`qos` The QoS value to set for each message. Has options 0, 1, 2. **Type**: `int` **Default**: `1` ### [](#retained)`retained` Set message as retained on the topic. **Type**: `bool` **Default**: `false` ### [](#retained_interpolated)`retained_interpolated` Override the value of `retained` with an interpolable value, this allows it to be dynamically set based on message contents. The value must resolve to either `true` or `false`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.59.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#urls)`urls[]` A list of URLs to connect to. Use the format `scheme://host:port`, where: - `scheme` is one of the following: `tcp`, `ssl`, `ws` - `host` is the IP address or hostname - `port` is the port on which the MQTT broker accepts connections If an item in the list contains commas, it is expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:1883" ``` ### [](#user)`user` A username to connect with. **Type**: `string` **Default**: `""` ### [](#will)`will` Set last will message in case of Redpanda Connect failure **Type**: `object` ### [](#will-enabled)`will.enabled` Whether to enable last will messages. **Type**: `bool` **Default**: `false` ### [](#will-payload)`will.payload` Set payload for last will message. **Type**: `string` **Default**: `""` ### [](#will-qos)`will.qos` Set QoS for last will message. Valid values are: 0, 1, 2. **Type**: `int` **Default**: `0` ### [](#will-retained)`will.retained` Set retained for last will message. **Type**: `bool` **Default**: `false` ### [](#will-topic)`will.topic` Set topic for last will message. **Type**: `string` **Default**: `""` ### [](#write_timeout)`write_timeout` The maximum amount of time to wait to write data before the attempt is abandoned. Requires version 3.58.0 or later. **Type**: `string` **Default**: `3s` ```yaml # Examples: write_timeout: 1s # --- write_timeout: 500ms ``` --- # Page 164: nanomsg **URL**: https://docs.redpanda.com/connect/components/outputs/nanomsg.md --- # nanomsg --- title: nanomsg latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/nanomsg page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nanomsg.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nanomsg.adoc description: Send messages over a Nanomsg socket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Send messages over a Nanomsg socket. Currently only PUSH and PUB sockets are supported. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" nanomsg: urls: [] # No default (required) bind: false socket_type: PUSH poll_timeout: 5s max_in_flight: 64 ``` ### Advanced ```yml output: label: "" nanomsg: urls: [] # No default (required) bind: false socket_type: PUSH poll_timeout: 5s max_in_flight: 64 ``` ## [](#fields)Fields ### [](#bind)`bind` Whether the URLs listed should be bind (otherwise they are connected to). **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#poll_timeout)`poll_timeout` The maximum period of time to wait for a message to send before the request is abandoned and reattempted. **Type**: `string` **Default**: `5s` ### [](#socket_type)`socket_type` The socket type to send with. **Type**: `string` **Default**: `PUSH` **Options**: `PUSH`, `PUB` ### [](#urls)`urls[]` A list of URLs to connect to, or to listen on when `bind` is `true`. If an item in the list contains commas, it is split into multiple URLs. A **host, such as `tcp://`**`:5555`, is replaced with `0.0.0.0`. **Type**: `array` --- # Page 165: nats_jetstream **URL**: https://docs.redpanda.com/connect/components/outputs/nats_jetstream.md --- # nats_jetstream --- title: nats_jetstream latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/nats_jetstream page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats_jetstream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats_jetstream.adoc description: Write messages to a NATS JetStream subject. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Write messages to a NATS JetStream subject. Introduced in version 3.46.0. #### Common ```yml output: label: "" nats_jetstream: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 1024 ``` #### Advanced ```yml output: label: "" nats_jetstream: urls: [] # No default (required) max_reconnects: 0 # No default (optional) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.1.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#metadata)`metadata` Specify which metadata values are added to messages as NATS headers, alongside any explicit `headers`. Requires version 4.22.0 or later. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` A subject to write to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: ${! meta("kafka_topic") } # --- subject: foo.${! json("meta.type") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 166: nats_kv **URL**: https://docs.redpanda.com/connect/components/outputs/nats_kv.md --- # nats_kv --- title: nats_kv latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/nats_kv page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats_kv.adoc description: Put messages in a NATS key-value bucket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Put messages into a NATS key-value bucket. Introduced in version 4.12.0. #### Common ```yml output: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) key: "" # No default (required) max_in_flight: 1024 ``` #### Advanced ```yml output: label: "" nats_kv: urls: [] # No default (required) max_reconnects: 0 # No default (optional) bucket: "" # No default (required) key: "" # No default (required) max_in_flight: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` The `key` field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), which lets you create a unique key for each message. ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#key)`key` The key for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: foo # --- key: foo.bar.baz # --- key: foo.${! json("meta.type") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 167: nats_stream **URL**: https://docs.redpanda.com/connect/components/outputs/nats_stream.md --- # nats_stream --- title: nats_stream latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/nats_stream page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats_stream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats_stream.adoc description: Publish to a NATS Stream subject. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Publish messages to a NATS Stream subject. #### Common ```yml output: label: "" nats_stream: urls: [] # No default (required) cluster_id: "" # No default (required) subject: "" # No default (required) client_id: "" max_in_flight: 64 ``` #### Advanced ```yml output: label: "" nats_stream: urls: [] # No default (required) max_reconnects: 0 # No default (optional) cluster_id: "" # No default (required) subject: "" # No default (required) client_id: "" max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` > ⚠️ **CAUTION: Deprecation notice** > > Deprecation notice > > The NATS Streaming Server is being deprecated. Critical bug fixes and security fixes will be applied until June of 2023. NATS-enabled applications requiring persistence should use [JetStream](https://docs.nats.io/nats-concepts/jetstream). ## [](#authentication)Authentication There are a number of Redpanda Connect components that use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#performance)Performance Send multiple messages in flight in parallel for improved performance. You can tune the number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#client_id)`client_id` The client ID to connect with. **Type**: `string` **Default**: `""` ### [](#cluster_id)`cluster_id` The cluster ID to publish to. **Type**: `string` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#subject)`subject` The subject to publish to. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 168: nats **URL**: https://docs.redpanda.com/connect/components/outputs/nats.md --- # nats --- title: nats latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/nats page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats.adoc description: Publish to an NATS subject. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Publish to an NATS subject. #### Common ```yml output: label: "" nats: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 64 ``` #### Advanced ```yml output: label: "" nats: urls: [] # No default (required) max_reconnects: 0 # No default (optional) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` This output interpolates functions within the subject field. For a full list of functions, see [configuration:interpolation.adoc#bloblang-queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#metadata)`metadata` Specify which metadata values are added to messages as NATS headers, alongside any explicit `headers`. Requires version 4.22.0 or later. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` The subject to publish to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 169: nsq **URL**: https://docs.redpanda.com/connect/components/outputs/nsq.md --- # nsq --- title: nsq latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/nsq page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nsq.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nsq.adoc description: Publish to an NSQ topic. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Publish to an NSQ topic. The `topic` field can be dynamically set using function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" nsq: nsqd_tcp_address: "" # No default (required) topic: "" # No default (required) user_agent: "" # No default (optional) max_in_flight: 64 ``` ### Advanced ```yml output: label: "" nsq: nsqd_tcp_address: "" # No default (required) topic: "" # No default (required) user_agent: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 ``` ## [](#fields)Fields ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#nsqd_tcp_address)`nsqd_tcp_address` The address of the target NSQD server. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#user_agent)`user_agent` A user agent to assume when connecting. **Type**: `string` --- # Page 170: ockam_kafka **URL**: https://docs.redpanda.com/connect/components/outputs/ockam_kafka.md --- # ockam_kafka --- title: ockam_kafka latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/ockam_kafka page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/ockam_kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/ockam_kafka.adoc description: Uses Ockam to encrypt and write end-to-end encrypted messages to a Kafka topic. page-git-created-date: "2024-11-25" page-git-modified-date: "2026-10-10" --- Uses [Ockam](https://docs.ockam.io/) to encrypt and write end-to-end encrypted messages to a Kafka topic. You can read encrypted messages using the `ockam_kafka` input or by [creating a Kafka Portal Inlet](https://command.ockam.io/manual/ockam-kafka-inlet-create.html) using Ockam Command. [Ockam Secure Channels](https://docs.ockam.io/reference/command/secure-channels) guarantee that consumers of topics only receive data from authenticated and authorized producers, and that the data has not been leaked or tampered with in-flight. Neither Kafka brokers, service providers nor other components can see or manipulate the messages. > 📝 **NOTE** > > You can use Ockam to encrypt whole messages, or specific fields in a message. Introduced in version 4.33.0. #### Common ```yml output: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_consumer: self route_to_consumer: /ip4/127.0.0.1/tcp/6262 encrypted_fields: [] ``` #### Advanced ```yml output: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) partitioner: "" # No default (optional) idempotent_write: true acks: all compression: snappy allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: "0" max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_consumer: self route_to_consumer: /ip4/127.0.0.1/tcp/6262 encrypted_fields: [] ``` ## [](#fields)Fields ### [](#allow)`allow` Use in conjunction with the `route_to_kafka_outlet` field to specify an access control policy for the Kafka Portal Outlet. For example, setting this value to `kafka_us_east` forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute `kafka_us_east=true`. **Type**: `string` **Default**: `self` ### [](#allow_consumer)`allow_consumer` Specify an access control policy for consumers. For example, setting this value to `orders_consumer` forces the consumer to present an Ockam credential, which confirms that the consumer has the attribute `orders_consumer=true`. **Type**: `string` **Default**: `self` ### [](#disable_content_encryption)`disable_content_encryption` Disables encryption of Kafka message payloads. If this value is set to `true`: - Only message payloads are unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled. - All other `ockam_kafka` inputs and outputs that use the same topic must also set this field to `true`. **Type**: `bool` **Default**: `false` ### [](#encrypted_fields)`encrypted_fields[]` The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted. Requires version 4.37.0 or later. **Type**: `array` **Default**: `[]` ### [](#enrollment_ticket)`enrollment_ticket` The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional). You can generate a new ticket using the [`ockam project ticket` command](https://command.ockam.io/manual/ockam-project-ticket.html). **Type**: `string` ### [](#identity_name)`identity_name` The name of the [Ockam identity](https://command.ockam.io/manual/ockam-identity.html) to use (optional). If an identity with this name does not exist, it is created. If this value is not provided, the default Ockam identity is used, and it is created if it does not exist. **Type**: `string` ### [](#kafka)`kafka` **Type**: `object` ### [](#kafka-acks)`kafka.acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#kafka-allow_auto_topic_creation)`kafka.allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. If set to `false`, the topic must already exist. Requires version 4.58.0 or later. **Type**: `bool` **Default**: `true` ### [](#kafka-batching)`kafka.batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#kafka-batching-byte_size)`kafka.batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#kafka-batching-check)`kafka.batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#kafka-batching-count)`kafka.batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#kafka-batching-period)`kafka.batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#kafka-batching-processors)`kafka.batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#kafka-broker_write_max_bytes)`kafka.broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. Requires version 4.35.0 or later. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#kafka-compression)`kafka.compression` The compression codec for produced record batches. A batch that compression does not make smaller is sent uncompressed, whichever codec you choose. **Type**: `string` **Default**: `snappy` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#kafka-idempotent_write)`kafka.idempotent_write` Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the `IDEMPOTENT_WRITE` permission on the `CLUSTER` resource. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally. **Type**: `bool` **Default**: `true` ### [](#kafka-key)`kafka.key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#kafka-max_buffered_bytes)`kafka.max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#kafka-max_buffered_records)`kafka.max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. Requires version 4.92.0 or later. **Type**: `int` **Default**: `10000` ### [](#kafka-max_in_flight)`kafka.max_in_flight` The maximum number of message batches to send in parallel at any given time. **Type**: `int` **Default**: `10` ### [](#kafka-max_in_flight_requests)`kafka.max_in_flight_requests` The maximum number of produce requests in flight per broker connection. While `idempotent_write` is enabled (the default) this must be `1`, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set `idempotent_write` to `false`, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s `max_in_flight` field, which counts message batches being written in parallel rather than produce requests on the wire. A value of `1` is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests. Requires version 4.92.0 or later. **Type**: `int` **Default**: `1` ### [](#kafka-max_message_bytes)`kafka.max_message_bytes` The maximum size of a produced record batch in bytes. A `MESSAGE_TOO_LARGE` error is returned if a batch exceeds this limit. This field maps to the `max.message.bytes` Kafka property. Ensure the Redpanda broker’s `kafka_batch_max_bytes` property is at least as large as this value, see [kafka\_batch\_max\_bytes](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_batch_max_bytes). **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#kafka-metadata)`kafka.metadata` Determines which metadata values are added to messages as headers. **Type**: `object` ### [](#kafka-metadata-include_patterns)`kafka.metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#kafka-metadata-include_prefixes)`kafka.metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#kafka-partition)`kafka.partition` An optional explicit partition to set for each message. This field is only relevant when the `partitioner` is set to `manual`. The provided interpolation string must be a valid integer. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#kafka-partitioner)`kafka.partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#kafka-record_delivery_timeout)`kafka.record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0s` ### [](#kafka-record_retries)`kafka.record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `int` **Default**: `0` ### [](#kafka-seed_brokers)`kafka.seed_brokers[]` The address of the Kafka broker that the Ockam node’s Kafka Outlet connects to. This field is only used, and is then required, when `route_to_kafka_outlet` is set to `self`. Only one address is used: the first list item, up to its first comma. If the list contains more than one item, a warning is logged. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" ``` ### [](#kafka-timeout)`kafka.timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#kafka-timestamp_ms)`kafka.timestamp_ms` An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both `timestamp` and `timestamp_ms`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.40.0 or later. **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#kafka-tls)`kafka.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#kafka-tls-client_certs)`kafka.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#kafka-tls-client_certs-cert)`kafka.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-cert_file)`kafka.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key)`kafka.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key_file)`kafka.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-password)`kafka.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#kafka-tls-enable_renegotiation)`kafka.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-enabled)`kafka.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-root_cas)`kafka.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#kafka-tls-root_cas_file)`kafka.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#kafka-tls-skip_cert_verify)`kafka.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#kafka-topic)`kafka.topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#route_to_consumer)`route_to_consumer` The route to the Kafka consumer. For example, `/project/default/service/forward_to_orders_consumer/secure/api` connects to a consumer exposed through a relay named `orders_consumer`. **Type**: `string` **Default**: `/ip4/127.0.0.1/tcp/6262` ### [](#route_to_kafka_outlet)`route_to_kafka_outlet` The route to reach the Kafka Portal Outlet of your Ockam portal. For example, `/project/default`. **Type**: `string` **Default**: `self` --- # Page 171: opensearch **URL**: https://docs.redpanda.com/connect/components/outputs/opensearch.md --- # opensearch --- title: opensearch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/opensearch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/opensearch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/opensearch.adoc description: Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. Introduced in version 4.25.0. Both the `id` and `index` fields can be dynamically set using function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" opensearch: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" opensearch: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] compress_request_body: false max_in_flight: 64 basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#examples)Examples ### [](#updating-documents)Updating Documents When [updating documents](https://opensearch.org/docs/latest/api-reference/document-apis/update-document/) the request body should contain a combination of a `doc`, `upsert`, and/or `script` fields at the top level, this should be done via mapping processors. ```yaml output: processors: - mapping: | meta id = this.id root.doc = this opensearch: urls: [ TODO ] index: foo id: ${! @id } action: update ``` ## [](#fields)Fields ### [](#action)`action` The action to take on the document. This field must resolve to one of the following action types: `index`, `update` or `delete`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#aws)`aws` Enables and customises connectivity to Amazon Elastic Service. **Type**: `object` ### [](#aws-credentials)`aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#aws-credentials-from_ec2_role)`aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#aws-credentials-id)`aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#aws-credentials-profile)`aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#aws-credentials-role)`aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#aws-credentials-role_external_id)`aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#aws-credentials-secret)`aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-credentials-token)`aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#aws-enabled)`aws.enabled` Whether to connect to Amazon Elastic Service. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#aws-tcp)`aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#aws-tcp-connect_timeout)`aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#aws-tcp-keep_alive)`aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#aws-tcp-keep_alive-count)`aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#aws-tcp-keep_alive-idle)`aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#aws-tcp-keep_alive-interval)`aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#aws-tcp-tcp_user_timeout)`aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#basic_auth)`basic_auth` Configure basic authentication credentials for connecting to OpenSearch. When enabled, these credentials are sent with each request to authenticate with the cluster. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to send the `username` and `password` credentials with each request. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#compress_request_body)`compress_request_body` Enable gzip compression of HTTP request bodies on the OpenSearch client. OpenSearch 2.x servers accept `Content-Encoding: gzip` natively; this is useful to reduce bandwidth on cross-cloud or cross-region bulk writes. **Type**: `bool` **Default**: `false` ### [](#id)`id` The ID for indexed messages. Use [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) to dynamically create a unique ID for each message. **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The OpenSearch index where messages are published. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` The ID of an optional ingest pipeline to preprocess incoming documents before they are indexed. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#routing)`routing` A custom routing value for each document, which determines the shard it is stored on. When empty, the document ID is used. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` --- # Page 172: otlp_grpc **URL**: https://docs.redpanda.com/connect/components/outputs/otlp_grpc.md --- # otlp_grpc --- title: otlp_grpc latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/otlp_grpc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/otlp_grpc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/otlp_grpc.adoc description: Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Introduced in version 4.78.0. Sends OpenTelemetry telemetry data to a remote collector via OTLP/gRPC protocol. Accepts batches of Redpanda OTEL v1 protobuf messages (spans, log records, or metrics) and converts them to OTLP format for transmission to OpenTelemetry collectors. #### Common ```yml output: label: "" otlp_grpc: endpoint: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml output: label: "" otlp_grpc: endpoint: "" # No default (required) headers: {} timeout: 30s compression: gzip tls: enabled: false skip_cert_verify: false cert_file: "" key_file: "" tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} max_in_flight: 64 ``` ## [](#input-format)Input format Expects messages in Redpanda OTEL v1 protobuf format with metadata: - `signal_type`: "trace", "log", or "metric" Each batch must contain messages of the same signal type. The entire batch is converted to a single OTLP export request and sent via gRPC. ## [](#authentication)Authentication Supports multiple authentication methods: - Bearer token authentication (via `auth_token` field) - OAuth v2 (via `oauth2` configuration block) > 📝 **NOTE** > > OAuth2 requires TLS to be enabled. ## [](#fields)Fields ### [](#compression)`compression` Compression type for gRPC requests. Options: 'gzip' or 'none'. **Type**: `string` **Default**: `gzip` **Options**: `gzip`, `none` ### [](#endpoint)`endpoint` The gRPC endpoint of the remote OTLP collector. **Type**: `string` ### [](#headers)`headers` A map of headers to add to the gRPC request metadata. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: X-Custom-Header: value traceparent: ${! tracing_span().traceparent } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#oauth2)`oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` Timeout for gRPC requests. **Type**: `string` **Default**: `30s` ### [](#tls)`tls` TLS configuration for gRPC client. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Skip certificate verification (insecure). **Type**: `bool` **Default**: `false` --- # Page 173: otlp_http **URL**: https://docs.redpanda.com/connect/components/outputs/otlp_http.md --- # otlp_http --- title: otlp_http latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/otlp_http page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/otlp_http.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/otlp_http.adoc description: Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Introduced in version 4.78.0. Sends OpenTelemetry telemetry data to a remote collector via OTLP/HTTP protocol. Accepts batches of Redpanda OTEL v1 protobuf messages (spans, log records, or metrics) and converts them to OTLP format for transmission to OpenTelemetry collectors. #### Common ```yml output: label: "" otlp_http: endpoint: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml output: label: "" otlp_http: endpoint: "" # No default (required) content_type: protobuf headers: {} timeout: 30s proxy_url: "" follow_redirects: false disable_http2: false tls: enabled: false skip_cert_verify: false cert_file: "" key_file: "" tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} max_in_flight: 64 ``` ## [](#input-format)Input format Expects messages in Redpanda OTEL v1 protobuf format with metadata: - `signal_type`: "trace", "log", or "metric" Each batch must contain messages of the same signal type. The entire batch is converted to a single OTLP export request and sent via HTTP POST. ## [](#endpoints)Endpoints The output automatically appends the signal type path to the base endpoint: - Traces: `{endpoint}/v1/traces` - Logs: `{endpoint}/v1/logs` - Metrics: `{endpoint}/v1/metrics` ## [](#content-types)Content types Supports two content types: - `protobuf` (default): `application/x-protobuf` - `json`: `application/json` ## [](#authentication)Authentication Supports multiple authentication methods: - Basic authentication - OAuth v1 - OAuth v2 - JWT ## [](#fields)Fields ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` Content type for HTTP requests. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `protobuf` **Options**: `protobuf`, `json` ### [](#disable_http2)`disable_http2` Whether or not to disable HTTP/2. **Type**: `bool` **Default**: `false` ### [](#endpoint)`endpoint` The HTTP endpoint of the remote OTLP collector (without the signal path). **Type**: `string` ### [](#follow_redirects)`follow_redirects` Whether to follow redirects, which are responses with 300-399 status codes, from the OTLP endpoint. When disabled, the output does not request the URL in the `Location` header, and the write fails with an unexpected status error. **Type**: `bool` **Default**: `false` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: X-Custom-Header: value traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` Timeout for HTTP requests. **Type**: `string` **Default**: `30s` ### [](#tls)`tls` TLS configuration for HTTP client. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Skip certificate verification (insecure). **Type**: `bool` **Default**: `false` --- # Page 174: pinecone **URL**: https://docs.redpanda.com/connect/components/outputs/pinecone.md --- # pinecone --- title: pinecone latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/pinecone page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/pinecone.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/pinecone.adoc description: Inserts items into a Pinecone index. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Inserts items into a Pinecone index. Introduced in version 4.31.0. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" pinecone: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" host: "" # No default (required) api_key: "" # No default (required) operation: upsert-vectors id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` ### Advanced ```yml output: label: "" pinecone: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) operation: upsert-vectors namespace: "" id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The Pinecone API key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#host)`host` The host for the Pinecone index. **Type**: `string` ### [](#id)`id` The ID for the index entry in Pinecone. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata_mapping)`metadata_mapping` An optional mapping of message to metadata in the Pinecone index entry. **Type**: `string` ```yaml # Examples: metadata_mapping: root = @ # --- metadata_mapping: root = metadata() # --- metadata_mapping: 'root = {"summary": this.summary, "foo": this.other_field}' ``` ### [](#namespace)`namespace` The namespace to write to - writes to the default namespace by default. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#operation)`operation` The operation to perform against the Pinecone index. **Type**: `string` **Default**: `upsert-vectors` **Options**: `update-vector`, `upsert-vectors`, `delete-vectors` ### [](#vector_mapping)`vector_mapping` The mapping to extract out the vector from the document. The result must be a floating point array. Required if not a delete operation. **Type**: `string` ```yaml # Examples: vector_mapping: root = this.embeddings_vector # --- vector_mapping: root = [1.2, 0.5, 0.76] ``` --- # Page 175: pulsar **URL**: https://docs.redpanda.com/connect/components/outputs/pulsar.md --- # pulsar --- title: pulsar latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/pulsar page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/pulsar.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/pulsar.adoc description: Write messages to an Apache Pulsar server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Write messages to an Apache Pulsar server. Introduced in version 3.43.0. #### Common ```yml output: label: "" pulsar: url: "" # No default (required) topic: "" # No default (required) tls: root_cas_file: "" key: "" ordering_key: "" max_in_flight: 64 ``` #### Advanced ```yml output: label: "" pulsar: url: "" # No default (required) topic: "" # No default (required) tls: root_cas_file: "" key: "" ordering_key: "" max_in_flight: 64 auth: oauth2: enabled: false audience: "" issuer_url: "" scope: "" private_key_file: "" token: enabled: false token: "" ``` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of Pulsar authentication methods. Requires version 3.60.0 or later. **Type**: `object` ### [](#auth-oauth2)`auth.oauth2` Parameters for Pulsar OAuth2 authentication. **Type**: `object` ### [](#auth-oauth2-audience)`auth.oauth2.audience` OAuth2 audience. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-enabled)`auth.oauth2.enabled` Whether OAuth2 is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-oauth2-issuer_url)`auth.oauth2.issuer_url` OAuth2 issuer URL. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-private_key_file)`auth.oauth2.private_key_file` The path to a file containing a private key. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-scope)`auth.oauth2.scope` OAuth2 scope to request. Requires version 4.29.0 or later. **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Parameters for Pulsar Token authentication. **Type**: `object` ### [](#auth-token-enabled)`auth.token.enabled` Whether Token Auth is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-token-token)`auth.token.token` Actual base64 encoded token. **Type**: `string` **Default**: `""` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#ordering_key)`ordering_key` The ordering key to publish messages with. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Specify the path to a custom CA certificate to trust the broker TLS service. Requires version 4.2.0 or later. **Type**: `object` ### [](#tls-root_cas_file)`tls.root_cas_file` The path of a PEM file containing the certificate authority certificates that the client trusts when it verifies the TLS certificate of the broker. Leave empty to trust the certificate authorities of the host system. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#topic)`topic` The topic to publish to. **Type**: `string` ### [](#url)`url` A URL to connect to. **Type**: `string` ```yaml # Examples: url: pulsar://localhost:6650 # --- url: pulsar://pulsar.us-west.example.com:6650 # --- url: pulsar+ssl://pulsar.us-west.example.com:6651 ``` --- # Page 176: pusher **URL**: https://docs.redpanda.com/connect/components/outputs/pusher.md --- # pusher --- title: pusher latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/pusher page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/pusher.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/pusher.adoc description: Output for publishing messages to Pusher API (https://pusher.com). page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Output for publishing messages to Pusher API ([https://pusher.com](https://pusher.com)) Introduced in version 4.3.0. #### Common ```yml output: label: "" pusher: batching: count: 0 byte_size: 0 period: "" check: "" channel: "" # No default (required) event: "" # No default (required) appId: "" # No default (required) key: "" # No default (required) secret: "" # No default (required) cluster: "" # No default (required) secure: true max_in_flight: 1 ``` #### Advanced ```yml output: label: "" pusher: batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) channel: "" # No default (required) event: "" # No default (required) appId: "" # No default (required) key: "" # No default (required) secret: "" # No default (required) cluster: "" # No default (required) secure: true max_in_flight: 1 ``` ## [](#fields)Fields ### [](#appid)`appId` Pusher app id **Type**: `string` ### [](#batching)`batching` maximum batch size is 10 (limit of the pusher library) **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#channel)`channel` Pusher channel to publish to. [Interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) can also be used **Type**: `string` ```yaml # Examples: channel: my_channel # --- channel: ${!json("id")} ``` ### [](#cluster)`cluster` Pusher cluster **Type**: `string` ### [](#event)`event` Event to publish to **Type**: `string` ### [](#key)`key` Pusher key **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to publish to Pusher in parallel. Each batch is sent as one trigger batch request. **Type**: `int` **Default**: `1` ### [](#secret)`secret` Pusher secret **Type**: `string` ### [](#secure)`secure` Enable SSL encryption **Type**: `bool` **Default**: `true` --- # Page 177: qdrant **URL**: https://docs.redpanda.com/connect/components/outputs/qdrant.md --- # qdrant --- title: qdrant latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/qdrant page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/qdrant.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/qdrant.adoc description: Adds items to a Qdrant collection. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Adds items to a [Qdrant](https://qdrant.tech/) collection. Introduced in version 4.33.0. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" qdrant: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" grpc_host: "" # No default (required) api_token: "" collection_name: "" # No default (required) id: "" # No default (required) vector_mapping: "" # No default (required) payload_mapping: root = {} ``` ### Advanced ```yml output: label: "" qdrant: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) grpc_host: "" # No default (required) api_token: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] collection_name: "" # No default (required) id: "" # No default (required) vector_mapping: "" # No default (required) payload_mapping: root = {} ``` ## [](#fields)Fields ### [](#api_token)`api_token` The Qdrant API token for authentication. Defaults to an empty string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#collection_name)`collection_name` The name of the collection in Qdrant. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#grpc_host)`grpc_host` The gRPC host of the Qdrant server. **Type**: `string` ```yaml # Examples: grpc_host: localhost:6334 # --- grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334 ``` ### [](#id)`id` The ID of the point to insert. Can be a UUID string or positive integer. **Type**: `string` ```yaml # Examples: id: root = "dc88c126-679f-49f5-ab85-04b77e8c2791" # --- id: root = 832 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#payload_mapping)`payload_mapping` An optional mapping of message to payload associated with the point. **Type**: `string` **Default**: `root = {}` ```yaml # Examples: payload_mapping: 'root = {"field": this.value, "field_2": 987}' # --- payload_mapping: root = metadata() ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#vector_mapping)`vector_mapping` The mapping to extract the vector from the document. **Type**: `string` ```yaml # Examples: vector_mapping: 'root = {"dense_vector": [0.352,0.532,0.754],"sparse_vector": {"indices": [23,325,532],"values": [0.352,0.532,0.532]}, "multi_vector": [[0.352,0.532],[0.352,0.532]]}' # --- vector_mapping: root = [1.2, 0.5, 0.76] # --- vector_mapping: root = this.vector # --- vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]] # --- vector_mapping: 'root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}}' # --- vector_mapping: 'root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]}' # --- vector_mapping: 'root = {"some_dense": [0.352,0.532,0.532,0.234]}' ``` --- # Page 178: questdb **URL**: https://docs.redpanda.com/connect/components/outputs/questdb.md --- # questdb --- title: questdb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/questdb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/questdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/questdb.adoc description: Pushes messages to a QuestDB table. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Pushes messages to a [QuestDB](https://questdb.io/docs/) table. Introduced in version 4.37.0. #### Common ```yml output: label: "" questdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" address: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) table: "" # No default (required) designated_timestamp_field: "" # No default (optional) designated_timestamp_unit: auto timestamp_string_fields: [] # No default (optional) timestamp_string_format: Jan _2 15:04:05.000000Z0700 symbols: [] # No default (optional) doubles: [] # No default (optional) error_on_empty_messages: false ``` #### Advanced ```yml output: label: "" questdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] address: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) retry_timeout: "" # No default (optional) request_timeout: "" # No default (optional) request_min_throughput: 0 # No default (optional) table: "" # No default (required) designated_timestamp_field: "" # No default (optional) designated_timestamp_unit: auto timestamp_string_fields: [] # No default (optional) timestamp_string_format: Jan _2 15:04:05.000000Z0700 symbols: [] # No default (optional) doubles: [] # No default (optional) error_on_empty_messages: false ``` > ❗ **IMPORTANT** > > Redpanda Data recommends enabling the dedupe feature on the QuestDB server. For more information about deploying, configuring, and using QuestDB, see the [QuestDB documentation](https://questdb.io/docs/). ## [](#performance)Performance For improved performance, this output sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches), using the `max_in_flight` field. You can configure batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#fields)Fields ### [](#address)`address` The host and port of the QuestDB server’s HTTP endpoint, excluding the protocol. **Type**: `string` ```yaml # Examples: address: localhost:9000 ``` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#designated_timestamp_field)`designated_timestamp_field` The name of the designated timestamp field in QuestDB. **Type**: `string` ### [](#designated_timestamp_unit)`designated_timestamp_unit` Units used for the designated timestamp field in QuestDB. **Type**: `string` **Default**: `auto` ### [](#doubles)`doubles[]` Columns that must be the `double` type, with `int` as the default. **Type**: `array` ### [](#error_on_empty_messages)`error_on_empty_messages` Mark a message as an error if it is empty after field validation. **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` The password to use for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#request_min_throughput)`request_min_throughput` The minimum expected throughput in bytes per second for HTTP requests. If the throughput is lower than this value, the connection times out. This output uses this value to calculate an additional timeout on top of the `request_timeout`. This setting is useful for large requests. Set it to `0` to disable this logic. **Type**: `int` ### [](#request_timeout)`request_timeout` The period of time to wait for a response from the QuestDB server, in addition to the timeout calculated from the `request_min_throughput` field. **Type**: `string` ### [](#retry_timeout)`retry_timeout` The period of time to continue retrying after a failed HTTP request. The interval between retries is an exponential backoff starting at 10 ms, and doubling after each failed attempt up to a maximum of 1 second. **Type**: `string` ### [](#symbols)`symbols[]` Columns that must be the `symbol` type. String values default to `string` types. **Type**: `array` ### [](#table)`table` The destination table in QuestDB. **Type**: `string` ```yaml # Examples: table: trades ``` ### [](#timestamp_string_fields)`timestamp_string_fields[]` String fields with textual timestamps. **Type**: `array` ### [](#timestamp_string_format)`timestamp_string_format` The timestamp format used when parsing timestamp string fields, specified as a layout for Go’s `time.Parse` function. **Type**: `string` **Default**: `Jan _2 15:04:05.000000Z0700` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#token)`token` The bearer token to use for authentication, which takes precedence over the basic authentication username and password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#username)`username` The username to use for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` --- # Page 179: redis_hash **URL**: https://docs.redpanda.com/connect/components/outputs/redis_hash.md --- # redis_hash --- title: redis_hash latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redis_hash page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_hash.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_hash.adoc description: Sets Redis hash objects using the HSET command. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sets Redis hash objects using the HSET command. The field `key` supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), allowing you to create a unique key for each message. The field `fields` allows you to specify an explicit map of field names to interpolated values, also evaluated per message of a batch: ```yaml output: redis_hash: url: tcp://localhost:6379 key: ${!json("id")} fields: topic: ${!meta("kafka_topic")} partition: ${!meta("kafka_partition")} content: ${!json("document.text")} ``` If the field `walk_metadata` is set to `true` then Redpanda Connect will walk all metadata fields of messages and add them to the list of hash fields to set. If the field `walk_json_object` is set to `true` then Redpanda Connect will walk each message as a JSON object, extracting keys and the string representation of their value and adds them to the list of hash fields to set. The order of hash field extraction is as follows: 1. Metadata (if enabled) 2. JSON object (if enabled) 3. Explicit fields Where latter stages will overwrite matching field names of a former stage. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" redis_hash: url: "" # No default (required) key: "" # No default (required) walk_metadata: false walk_json_object: false fields: {} max_in_flight: 64 ``` ### Advanced ```yml output: label: "" redis_hash: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) walk_metadata: false walk_json_object: false fields: {} max_in_flight: 64 ``` ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#fields-2)`fields` A map of key/value pairs to set as hash fields. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ### [](#key)`key` The key of the hash to set fields on for each message. Use function interpolations to create a unique key per message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: ${! @.kafka_key } # --- key: ${! this.doc.id } # --- key: ${! counter() } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#walk_json_object)`walk_json_object` Whether to walk each message as a JSON object and add each key/value pair to the list of hash fields to set. **Type**: `bool` **Default**: `false` ### [](#walk_metadata)`walk_metadata` Whether all metadata fields of messages should be walked and added to the list of hash fields to set. **Type**: `bool` **Default**: `false` --- # Page 180: redis_list **URL**: https://docs.redpanda.com/connect/components/outputs/redis_list.md --- # redis_list --- title: redis_list latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redis_list page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_list.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_list.adoc description: Pushes messages onto the end of a Redis list (which is created if it doesn't already exist) using the RPUSH command. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Pushes messages onto the end of a Redis list (which is created if it doesn’t already exist) using the RPUSH command. The field `key` supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries), allowing you to create a unique key for each message. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" redis_list: url: "" # No default (required) key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" redis_list: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) command: rpush ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command used to push elements to the Redis list Requires version 4.22.0 or later. **Type**: `string` **Default**: `rpush` **Options**: `rpush`, `lpush` ### [](#key)`key` The key of the list to push each message to. Function interpolations can be used to create a different key per message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: some_list # --- key: ${! @.kafka_key } # --- key: ${! this.doc.id } # --- key: ${! counter() } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 181: redis_pubsub **URL**: https://docs.redpanda.com/connect/components/outputs/redis_pubsub.md --- # redis_pubsub --- title: redis_pubsub latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redis_pubsub page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_pubsub.adoc description: Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received. This output will interpolate functions within the channel field, you can find a list of functions [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" redis_pubsub: url: "" # No default (required) channel: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" redis_pubsub: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] channel: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#channel)`channel` The channel to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 182: redis_streams **URL**: https://docs.redpanda.com/connect/components/outputs/redis_streams.md --- # redis_streams --- title: redis_streams latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redis_streams page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_streams.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_streams.adoc description: Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn't already exist) using the XADD command. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn’t already exist) using the XADD command. It’s possible to specify a maximum length of the target stream by setting it to a value greater than 0, in which case this cap is applied only when Redis is able to remove a whole macro node, for efficiency. Redis stream entries are key/value pairs, as such it is necessary to specify the key to be set to the body of the message. All metadata fields of the message will also be set as key/value pairs, if there is a key collision between a metadata item and the body then the body takes precedence. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" redis_streams: url: "" # No default (required) stream: "" # No default (required) id: "*" body_key: body max_length: 0 max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" redis_streams: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] stream: "" # No default (required) id: "*" body_key: body max_length: 0 max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#body_key)`body_key` A key to set the raw body of the message to. **Type**: `string` **Default**: `body` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#id)`id` The entry ID for the stream message. Allows function interpolations. When set to `*` (the default), Redis auto-generates a unique ID based on the current time. Set a custom ID to control message ordering, for example to replay messages in upstream order. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.84.0 or later. **Type**: `string` **Default**: `*` ```yaml # Examples: id: "*" # --- id: ${! @redis_stream } # --- id: ${! this.id } # --- id: ${! counter() }-0 ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_length)`max_length` When greater than zero enforces a rough cap on the length of the target stream. **Type**: `int` **Default**: `0` ### [](#metadata)`metadata` Specify criteria for which metadata values are included in the message body. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#stream)`stream` The stream to add messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 183: redpanda_common **URL**: https://docs.redpanda.com/connect/components/outputs/redpanda_common.md --- # redpanda_common --- title: redpanda_common latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redpanda_common page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redpanda_common.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redpanda_common.adoc description: Sends data to a Redpanda (Kafka) broker, using credentials defined in a common top-level redpanda config block. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) components. Sends data to a Redpanda (Kafka) broker, using credentials from a common `redpanda` configuration block. To avoid duplicating Redpanda cluster credentials in your `redpanda_common` input, output, or any other components in your data pipeline, you can use a single [`redpanda` configuration block](https://docs.redpanda.com/connect/components/redpanda/about/). For more details, see the [Pipeline example](#pipeline-example). Introduced in version 4.39.0. > 📝 **NOTE** > > If you need to move topic data between Redpanda clusters or other Apache Kafka clusters, consider using the [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [output](https://docs.redpanda.com/connect/components/outputs/redpanda/) instead. #### Common ```yml output: label: "" redpanda_common: topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" redpanda_common: topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `10` ### [](#metadata)`metadata` Determines which metadata values are added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#partition)`partition` An optional explicit partition to set for each message. This field is only relevant when the `partitioner` is set to `manual`. The provided interpolation string must be a valid integer. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#timestamp_ms)`timestamp_ms` An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both `timestamp` and `timestamp_ms`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.40.0 or later. **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 184: redpanda_migrator **URL**: https://docs.redpanda.com/connect/components/outputs/redpanda_migrator.md --- # redpanda_migrator --- title: redpanda_migrator latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redpanda_migrator page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redpanda_migrator.adoc description: A specialised Kafka producer for comprehensive data migration between Apache Kafka and Redpanda clusters. page-git-created-date: "2024-10-02" page-git-modified-date: "2026-10-14" --- Migrates topics, schemas, and consumer groups between Kafka and Redpanda clusters. > ❗ **IMPORTANT** > > Pair this output with a [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) in the same pipeline. Introduced in version 4.37.0. The following shows all available configuration fields and their defaults. #### Common ```yml output: label: "" redpanda_migrator: seed_brokers: [] # No default (required) schema_registry: url: "" # No default (required) timeout: 5s enabled: true interval: 5m include: [] # No default (optional) exclude: [] # No default (optional) subject: "" # No default (optional) versions: all include_deleted: false translate_ids: false normalize: false strict: false max_parallel_http_requests: 10 consumer_groups: enabled: true interval: 1m fetch_timeout: 10s include: [] # No default (optional) exclude: [] # No default (optional) only_empty: false topic: ${! @kafka_topic } topic_replication_factor: 0 # No default (optional) sync_topic_acls: false headers: {} # No default (optional) max_in_flight: 10 ``` #### Advanced ```yml output: label: "" redpanda_migrator: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s partitioner: "" # No default (optional) idempotent_write: true acks: all compression: snappy allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: "0" max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} enabled: true interval: 5m include: [] # No default (optional) exclude: [] # No default (optional) subject: "" # No default (optional) versions: all include_deleted: false translate_ids: false normalize: false strict: false max_parallel_http_requests: 10 consumer_groups: enabled: true interval: 1m fetch_timeout: 10s include: [] # No default (optional) exclude: [] # No default (optional) only_empty: false topic: ${! @kafka_topic } topic_replication_factor: 0 # No default (optional) sync_topic_interval: 5m sync_topic_acls: false serverless: false headers: {} # No default (optional) provenance_header: redpanda-migrator-provenance offset_header: redpanda-migrator-offset max_in_flight: 10 ``` ## [](#requirements)Requirements When the destination cluster enforces ACLs, the destination principal needs permission to create topics and add partitions, not only to produce records. Grant these ACLs at minimum: - Topic `CREATE`, `WRITE`, `ALTER`, and `DESCRIBE_CONFIGS`. Cluster `CREATE` also authorizes topic creation. - When consumer group migration is enabled, consumer group `READ`. - When `sync_topic_acls` is enabled, cluster `ALTER`. For the source-principal ACLs and full details, see [Required permissions](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/#required-permissions). ## [](#multiple-migrator-pairs)Multiple migrator pairs Each migrator pair requires a unique `label`. Set the same label value on both the input and output within a pair. Labels must match exactly; mismatched labels prevent the input and output from coordinating. ## [](#performance-tuning)Performance tuning For high-throughput workloads, adjust the following settings: On this output: - `max_in_flight`: Set to the total number of partitions being migrated. Higher values provide no benefit beyond the partition count. On the paired [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/#performance-tuning-for-high-throughput): - `partition_buffer_bytes`: Set to `2MB` to increase the per-partition buffer. - `max_yield_batch_bytes`: Set to `1MB` to yield larger batches. ## [](#synchronization-details)Synchronization details ### [](#topics)Topics - Topic names carry over from the source by default. Use the `topic` field with interpolation to rename topics at the destination. - The migrator creates each topic at the destination with the same partition count as the source. - Replication factor defaults to the source value. Set `topic_replication_factor` to override it. - The migrator copies a serverless-aware subset of topic configuration keys. - ACL replication is optional (`sync_topic_acls`): - The migrator does not copy `ALLOW WRITE` ACL entries. - The migrator downgrades `ALLOW ALL` ACL entries to `ALLOW READ`. - Resource pattern type and host filters carry over. ### [](#schema-registry)Schema Registry - Syncs once at startup, then periodically based on `schema_registry.interval` (default: every 5 minutes). Set `schema_registry.interval: 0s` for a one-time sync only. - Use include and exclude regex patterns to filter which subjects are migrated. - Use the `subject` field with interpolation to rename subjects at the destination. - Schema versions: `latest` (most recent version only) or `all` (full history). Defaults to `all`. - Include soft-deleted subjects with `schema_registry.include_deleted`. - The migrator can translate schema IDs to new destination IDs or preserve them as-is. See `schema_registry.translate_ids`. - Schema normalization is optional (`schema_registry.normalize`). - Compatibility settings carry over per-subject. Schema metadata and rules do not carry over in Serverless mode. ### [](#consumer-groups)Consumer groups - Sync periodically based on `consumer_groups.interval` (default: every 1 minute). - Use include and exclude regex patterns to filter which groups are migrated. - By default, all groups except those in `Dead` state migrate. Set `consumer_groups.only_empty: true` to migrate only `Empty` state groups. - The migrator translates consumer offsets using timestamps. This is approximate and may be imprecise when multiple records share the same timestamp. - Offsets only move forward and never rewind at the destination. - Source and destination must have matching partition counts for each migrated group. ## [](#how-it-works)How it works Each migration component synchronizes on a different schedule: - **Topics**: Sync from source on startup and every 5 minutes by default, including source topics that have no current data (for example, after retention cleanup). Configure the sync interval with [`sync_topic_interval`](#sync_topic_interval), or set it to `0s` to disable periodic sync. When periodic sync is disabled, topics are still created on demand when the first message arrives. - **Schema Registry**: Syncs at startup, then periodically as configured. - **Consumer groups**: Sync in the background, filtered to the topics being migrated. ## [](#guarantees)Guarantees The migrator upholds the following guarantees: - Creates each destination topic with the intended partition count and replication factor. - Never overwrites existing destination topics and logs any partition count mismatches. - Consumer group offsets never rewind. - ACL replication never grants write access at the destination. ## [](#limitations)Limitations - The destination cluster’s Schema Registry must be in `READWRITE` or `IMPORT` mode. - Offset translation is best-effort. - Consumer group migration requires identical partition counts at source and destination. ## [](#metrics)Metrics | Metric Name | Type | Labels | Description | | --- | --- | --- | --- | | Topic migration | | | | | redpanda_migrator_topics_created_total | counter | | Total topics created on destination | | redpanda_migrator_topic_create_errors_total | counter | | Topic creation errors | | redpanda_migrator_topic_create_latency_ns | timer | | Topic creation latency (ns) | | Schema Registry migration | | | | | redpanda_migrator_sr_schemas_created_total | counter | | Schemas created in destination registry | | redpanda_migrator_sr_schema_create_errors_total | counter | | Schema creation errors | | redpanda_migrator_sr_schema_create_latency_ns | timer | | Schema creation latency (ns) | | redpanda_migrator_sr_compatibility_updates_total | counter | | Compatibility level updates applied | | redpanda_migrator_sr_compatibility_update_errors_total | counter | | Compatibility update errors | | redpanda_migrator_sr_compatibility_update_latency_ns | timer | | Compatibility update latency (ns) | | Consumer group migration | | | | | redpanda_migrator_cg_offsets_translated_total | counter | group | Offsets translated per consumer group | | redpanda_migrator_cg_offset_translation_errors_total | counter | group | Offset translation errors per group | | redpanda_migrator_cg_offset_translation_latency_ns | timer | group | Offset translation latency per group (ns) | | redpanda_migrator_cg_offsets_committed_total | counter | group | Offsets committed per consumer group | | redpanda_migrator_cg_offset_commit_errors_total | counter | group | Offset commit errors per group | | redpanda_migrator_cg_offset_commit_latency_ns | timer | group | Offset commit latency per group (ns) | | Consumer lag | | | | | redpanda_lag | gauge | topic, partition | Current consumer lag in messages for each topic partition. Shows difference between high water mark and current consumer position. | ## [](#examples)Examples ### [](#basic-migration)Basic migration Migrate topics, schemas and consumer groups from source to destination. ```yaml input: redpanda_migrator: seed_brokers: ["source:9092"] topics: ["orders", "payments"] consumer_group: "migration" output: redpanda_migrator: seed_brokers: ["destination:9092"] # Write to the same topic name topic: ${! metadata("kafka_topic") } schema_registry: url: "http://dest-registry:8081" translate_ids: true consumer_groups: interval: 1m ``` ### [](#migration-to-redpanda-serverless)Migration to Redpanda Serverless Migrate from Confluent/Kafka to Redpanda Cloud serverless cluster with authentication. ```yaml input: redpanda_migrator: seed_brokers: ["source-kafka:9092"] regexp_topics_include: - '.' regexp_topics_exclude: - '^_' consumer_group: "migrator_cg" schema_registry: url: "http://source-registry:8081" output: redpanda_migrator: seed_brokers: ["serverless-cluster.redpanda.com:9092"] tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "migrator" password: "migrator" schema_registry: url: "https://serverless-cluster.redpanda.com:8081" basic_auth: enabled: true username: "migrator" password: "migrator" translate_ids: true consumer_groups: exclude: - "migrator_cg" # Exclude the migration consumer group itself serverless: true # Enable serverless mode for restricted configurations ``` ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. If set to `false`, the topic must already exist. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` The compression codec for produced record batches. A batch that compression does not make smaller is sent uncompressed, whichever codec you choose. **Type**: `string` **Default**: `snappy` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_groups)`consumer_groups` **Type**: `object` ### [](#consumer_groups-enabled)`consumer_groups.enabled` Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed. **Type**: `bool` **Default**: `true` ### [](#consumer_groups-exclude)`consumer_groups.exclude[]` Regular expressions for consumer groups to exclude from offset migration. Takes precedence over include patterns. Useful for excluding system or temporary groups. **Type**: `array` ```yaml # Examples: exclude: - .*-test - .*-temp - connect-.* # --- exclude: - dev-.* - local-.* ``` ### [](#consumer_groups-fetch_timeout)`consumer_groups.fetch_timeout` Maximum time to wait for data when fetching records for timestamp-based offset translation. Increase for clusters with low message throughput. **Type**: `string` **Default**: `10s` ```yaml # Examples: fetch_timeout: "1s # Fast clusters" # --- fetch_timeout: "10s # Slower clusters" ``` ### [](#consumer_groups-include)`consumer_groups.include[]` Regular expressions for consumer groups to include in offset migration. If empty, all groups are included (unless excluded). **Type**: `array` ```yaml # Examples: include: - prod-.* - staging-.* # --- include: - app-.* - service-.* ``` ### [](#consumer_groups-interval)`consumer_groups.interval` How often to synchronise consumer group offsets. Regular syncing helps maintain offset accuracy during ongoing migration. **Type**: `string` **Default**: `1m` ```yaml # Examples: interval: "0s # Disabled" # --- interval: "30s # Sync every 30 seconds" # --- interval: "5m # Sync every 5 minutes" ``` ### [](#consumer_groups-only_empty)`consumer_groups.only_empty` Whether to only migrate Empty consumer groups. When false (default), all statuses except Dead are included; when true, only Empty groups are migrated. **Type**: `bool` **Default**: `false` ### [](#headers)`headers` Custom headers to add to migrated records, keyed by header name with interpolated string values. Useful for injecting metadata such as processing timestamps or latency measurements that should surface as header values on the destination cluster. A custom header name that collides with `provenance_header` or `offset_header` is ignored, so those migration-critical headers are always protected. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.102.0 or later. **Type**: `object` ```yaml # Examples: headers: x-migration-latency-ms: ${! timestamp_unix_milli() - meta("kafka_timestamp_ms") } x-migration-processed-at: ${! timestamp_unix_milli() } ``` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the `IDEMPOTENT_WRITE` permission on the `CLUSTER` resource. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally. **Type**: `bool` **Default**: `true` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. Requires version 4.92.0 or later. **Type**: `int` **Default**: `10000` ### [](#max_in_flight)`max_in_flight` The maximum number of batches to send in parallel at any given time. Increase this value to improve throughput during migration. For optimal performance, set this to the total number of partitions being migrated in parallel (up to all partitions in the cluster). Setting it higher than the number of consumed partitions provides no additional benefit. For example, if you are migrating 100 partitions, set `max_in_flight: 100` for maximum throughput. **Type**: `int` **Default**: `10` ```yaml # Examples: max_in_flight: "64 # For a cluster with 64 partitions" # --- max_in_flight: "128 # For multiple topics with combined 128 partitions" ``` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. While `idempotent_write` is enabled (the default) this must be `1`, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set `idempotent_write` to `false`, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s `max_in_flight` field, which counts message batches being written in parallel rather than produce requests on the wire. A value of `1` is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests. Requires version 4.92.0 or later. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum size of a produced record batch in bytes. A `MESSAGE_TOO_LARGE` error is returned if a batch exceeds this limit. This field maps to the `max.message.bytes` Kafka property. Ensure the Redpanda broker’s `kafka_batch_max_bytes` property is at least as large as this value, see [kafka\_batch\_max\_bytes](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_batch_max_bytes). **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#offset_header)`offset_header` The name of a message header to add to migrated records. This header contains the source offset, enabling exact consumer group offset translation during migration. If this field is empty, no offset header is added and exact offset translation is disabled. Consumer groups are still migrated using timestamp-based positioning, which works well for most cases but may be imprecise for consumer groups in the `Empty` state when multiple records share the same timestamp (timestamps have millisecond resolution). Exact offset translation only applies to consumer groups in the `Empty` state, which have no active members. Other groups always use timestamp-based positioning. The default value enables exact offset translation. This header is only added when consumer group migration is enabled. Requires version 4.71.0 or later. **Type**: `string` **Default**: `redpanda-migrator-offset` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#provenance_header)`provenance_header` Header name to add to migrated records indicating their source cluster. When set, each migrated record that does not already carry this header receives it, with the source cluster’s ID (from the cluster metadata) as the value. Downstream systems can use the header to track record origins. A record that already carries the header keeps its existing value. A record whose header value is the destination cluster’s ID is skipped, so it is not sent back to the cluster it came from. A record whose header value is empty or equal to the source cluster’s ID causes an error. If empty, no provenance header is added and no provenance checks run. Requires version 4.69.0 or later. **Type**: `string` **Default**: `redpanda-migrator-provenance` ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-enabled)`schema_registry.enabled` Whether schema registry migration is enabled. When disabled, no schema operations are performed. **Type**: `bool` **Default**: `true` ### [](#schema_registry-exclude)`schema_registry.exclude[]` Regular expressions for schema subjects to exclude from migration. Takes precedence over include patterns. Note: the migrator consumer group is always ignored. **Type**: `array` ```yaml # Examples: exclude: - .*-test - .*-temp # --- exclude: - dev-.* - local-.* ``` ### [](#schema_registry-include)`schema_registry.include[]` Regular expressions for schema subjects to include in migration. If empty, all subjects are included (unless excluded). Note: the migrator consumer group is always ignored. **Type**: `array` ```yaml # Examples: include: - prod-.* - staging-.* # --- include: - user-.* - order-.* ``` ### [](#schema_registry-include_deleted)`schema_registry.include_deleted` Whether to include soft-deleted schemas in migration. Useful for complete migration but may not be supported by all schema registries. **Type**: `bool` **Default**: `false` ### [](#schema_registry-interval)`schema_registry.interval` How often to synchronise schema registry subjects. Subjects that fail to sync are logged and retried on the next sync. Set to 0s for one-time sync at startup only, in which case only the subjects that failed are retried, with backoff, until they sync. With translate\_ids enabled, a record encoded with a schema that failed to sync blocks the output until the schema is synced. **Type**: `string` **Default**: `5m` ```yaml # Examples: interval: "0s # One-time sync only" # --- interval: "5m # Sync every 5 minutes" # --- interval: "30m # Sync every 30 minutes" ``` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-max_parallel_http_requests)`schema_registry.max_parallel_http_requests` Maximum number of parallel HTTP requests to the schema registry. Controls concurrency when syncing multiple schemas. Requires version 4.80.0 or later. **Type**: `int` **Default**: `10` ### [](#schema_registry-normalize)`schema_registry.normalize` Whether to normalize schemas when creating them in the destination registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-strict)`schema_registry.strict` Error on unknown schema IDs. Only relevant when translate\_ids is true. When false (default), unknown schema IDs are passed through unchanged, allowing migration of topics with mixed message formats. IDs of schemas that failed to sync to the destination are always rejected, regardless of this setting, and the write is retried until a later sync registers the schema, or the source no longer has it. Since the output writes in order, this blocks the output until then. Note: messages with 0-byte prefixes (for example, protobuf) cannot be distinguished from schema registry headers and may fail when strict is enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-subject)`schema_registry.subject` Template for transforming subject names during migration. Use interpolation to rename subjects systematically. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: prod_${! metadata("schema_registry_subject") } # --- subject: ${! metadata("schema_registry_subject") | replace("dev_", "prod_") } ``` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for schema registry requests. Requires version 4.72.0 or later. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-translate_ids)`schema_registry.translate_ids` Whether to translate schema IDs during migration. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. Required for schema migration functionality. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 # --- url: https://schema-registry.example.com:8081 ``` ### [](#schema_registry-versions)`schema_registry.versions` Which schema versions to migrate. 'latest' migrates only the current version, 'all' migrates complete version history for better compatibility. **Type**: `string` **Default**: `all` **Options**: `latest`, `all` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#serverless)`serverless` Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by serverless environments. **Type**: `bool` **Default**: `false` ### [](#sync_topic_acls)`sync_topic_acls` Whether to synchronise topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts. **Type**: `bool` **Default**: `false` ### [](#sync_topic_interval)`sync_topic_interval` How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow. Set to 0s to disable periodic sync (topics are still created on first message). Requires version 4.82.0 or later. **Type**: `string` **Default**: `5m` ```yaml # Examples: sync_topic_interval: "0s # Disable periodic sync" # --- sync_topic_interval: "1m # Sync every minute" # --- sync_topic_interval: "5m # Sync every 5 minutes" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to write messages to. To derive destination topic names from source topics, use interpolation. The source topic name is available in the `kafka_topic` metadata field. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${! @kafka_topic }` ```yaml # Examples: topic: prod_${! @kafka_topic } ``` ### [](#topic_replication_factor)`topic_replication_factor` The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes. **Type**: `int` ```yaml # Examples: topic_replication_factor: "3" # --- topic_replication_factor: "1 # For single-node clusters" ``` --- # Page 185: redpanda **URL**: https://docs.redpanda.com/connect/components/outputs/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/redpanda page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redpanda.adoc description: A Kafka output using the Franz Kafka client library. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-14" --- Sends message data to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input. Introduced in version 4.39.0. To share one connection across several `redpanda` components, omit `seed_brokers` and configure the top-level [`redpanda` block](https://docs.redpanda.com/connect/components/redpanda/about/). #### Common ```yml output: label: "" redpanda: seed_brokers: [] # No default (optional) topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 256 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" redpanda: seed_brokers: [] # No default (optional) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 256 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) inject_tracing_map: "" # No default (optional) partitioner: "" # No default (optional) idempotent_write: true acks: all compression: snappy allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: "0" max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s ``` ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. If set to `false`, the topic must already exist. Requires version 4.58.0 or later. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Optional explicit batching policy for the output. Note that when batches are formed at the input level they can be expanded by this policy, but not contracted. When consuming data from a Redpanda input it is recommended to tune batches from the input config via the `max_yield_batch_bytes` field, or the `unordered_processing.batching` field if appropriate. Requires version 4.75.0 or later. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` The compression codec for produced record batches. A batch that compression does not make smaller is sent uncompressed, whichever codec you choose. **Type**: `string` **Default**: `snappy` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.53.0 or later. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the `IDEMPOTENT_WRITE` permission on the `CLUSTER` resource. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally. **Type**: `bool` **Default**: `true` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.85.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. Requires version 4.92.0 or later. **Type**: `int` **Default**: `10000` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to send in parallel at any given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `256` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. While `idempotent_write` is enabled (the default) this must be `1`, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set `idempotent_write` to `false`, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s `max_in_flight` field, which counts message batches being written in parallel rather than produce requests on the wire. A value of `1` is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests. Requires version 4.92.0 or later. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum size of a produced record batch in bytes. A `MESSAGE_TOO_LARGE` error is returned if a batch exceeds this limit. This field maps to the `max.message.bytes` Kafka property. Ensure the Redpanda broker’s `kafka_batch_max_bytes` property is at least as large as this value, see [kafka\_batch\_max\_bytes](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_batch_max_bytes). **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata)`metadata` Determines which metadata values are added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#partition)`partition` An optional explicit partition to set for each message. This field is only relevant when the `partitioner` is set to `manual`. The provided interpolation string must be a valid integer. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. Requires version 4.53.0 or later. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. If you omit this field, this component ignores the connection fields set on it, including `tls` and `sasl`, and takes its entire connection configuration from the top-level `redpanda` block. In Redpanda Cloud, the platform sets the top-level `redpanda` block for pipeline logs and status reporting, and its credentials can’t access your topics, so always set this field in Redpanda Cloud. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#timestamp_ms)`timestamp_ms` An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both `timestamp` and `timestamp_ms`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.40.0 or later. **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ## [](#examples)Examples ### [](#simple-common-output)Simple Common Output Data is generated and written to a topic `bar`, targeting the cluster configured within the `redpanda` block at the bottom. This lets you configure TLS and SASL once and share them across multiple inputs and outputs. ```yaml input: generate: interval: 1s mapping: 'root.name = fake("name")' pipeline: processors: - mutation: | root.id = uuid_v4() root.loud_name = this.name.uppercase() output: redpanda: topic: bar key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` --- # Page 186: reject_errored **URL**: https://docs.redpanda.com/connect/components/outputs/reject_errored.md --- # reject_errored --- title: reject_errored latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/reject_errored page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/reject_errored.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/reject_errored.adoc description: Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output. Introduced in version 4.27.0. The routing of messages rejected by this output depends on the type of input it came from. For inputs that support propagating nacks upstream such as AMQP or NATS the message will be nacked. However, for inputs that are sequential such as files or Kafka the messages will simply be reprocessed from scratch. #### Common ```yml output: label: "" reject_errored: {} ``` #### Advanced ```yml output: label: "" reject_errored: {} ``` ## [](#examples)Examples ### [](#rejecting-failed-messages)Rejecting Failed Messages The most straight forward use case for this output type is to nack messages that have failed their processing steps. In this example our mapping might fail, in which case the messages that failed are rejected and will be nacked by our input: ```yaml input: nats_jetstream: urls: [ nats://127.0.0.1:4222 ] subject: foos.pending pipeline: processors: - mutation: 'root.age = this.fuzzy.age.int64()' output: reject_errored: nats_jetstream: urls: [ nats://127.0.0.1:4222 ] subject: foos.processed ``` ### [](#dlqing-failed-messages)DLQing Failed Messages Another use case for this output is to send failed messages straight into a dead-letter queue. You use it within a [fallback output](https://docs.redpanda.com/connect/components/outputs/fallback/) that allows you to specify where these failed messages should go to next. ```yaml pipeline: processors: - mutation: 'root.age = this.fuzzy.age.int64()' output: fallback: - reject_errored: http_client: url: http://foo:4195/post/might/become/unreachable retries: 3 retry_period: 1s - http_client: url: http://bar:4196/somewhere/else retries: 3 retry_period: 1s ``` --- # Page 187: reject **URL**: https://docs.redpanda.com/connect/components/outputs/reject.md --- # reject --- title: reject latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/reject page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/reject.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/reject.adoc description: Rejects all messages, treating them as though the output destination failed to publish them. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Rejects all messages, treating them as though the output destination failed to publish them. The routing of messages after this output depends on the type of input it came from. For inputs that support propagating nacks upstream such as AMQP or NATS the message will be nacked. However, for inputs that are sequential such as files or Kafka the messages will simply be reprocessed from scratch. To learn when this output could be useful, see \[the [Examples](#examples). #### Common ```yml output: label: "" reject: "" ``` #### Advanced ```yml output: label: "" reject: "" ``` ## [](#examples)Examples ### [](#rejecting-failed-messages)Rejecting Failed Messages This input is particularly useful for routing messages that have failed during processing, where instead of routing them to some sort of dead letter queue we wish to push the error upstream. We can do this with a switch broker: ```yaml output: switch: retry_until_success: false cases: - check: '!errored()' output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - output: reject: "processing failed due to: ${! error() }" ``` --- # Page 188: resource **URL**: https://docs.redpanda.com/connect/components/outputs/resource.md --- # resource --- title: resource latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/resource page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/resource.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/resource.adoc description: Resource is an output type that channels messages to a resource output, identified by its name. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Resource is an output type that channels messages to a resource output, identified by its name. ## Common ```yml output: resource: "" ``` ## Advanced ```yml output: resource: "" ``` Resources allow you to tidy up deeply nested configs. For example, the config: ```yaml output: broker: pattern: fan_out outputs: - kafka: addresses: [ TODO ] topic: foo - gcp_pubsub: project: bar topic: baz ``` Could also be expressed as: ```yaml output: broker: pattern: fan_out outputs: - resource: foo - resource: bar output_resources: - label: foo kafka: addresses: [ TODO ] topic: foo - label: bar gcp_pubsub: project: bar topic: baz ``` You can find out more about resources in [Resources](https://docs.redpanda.com/connect/configuration/resources/) --- # Page 189: retry **URL**: https://docs.redpanda.com/connect/components/outputs/retry.md --- # retry --- title: retry latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/retry page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/retry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/retry.adoc description: Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or max elapsed time fields are non-zero, either is reached. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or max elapsed time fields are non-zero, either is reached. All messages in Redpanda Connect are always retried on an output error, but this would usually involve propagating the error back to the source of the message, whereby it would be reprocessed before reaching the output layer once again. This output type is useful whenever we wish to avoid reprocessing a message on the event of a failed send. We might, for example, have a deduplication processor that we want to avoid reapplying to the same message more than once in the pipeline. Rather than retrying the same output you may wish to retry the send using a different output target (a dead letter queue). In which case you should instead use the [`fallback`](https://docs.redpanda.com/connect/components/outputs/fallback/) output type. #### Common ```yml output: label: "" retry: output: {} # No default (required) ``` #### Advanced ```yml output: label: "" retry: max_retries: 0 backoff: initial_interval: 500ms max_interval: 3s max_elapsed_time: 0s output: {} # No default (required) ``` ## [](#fields)Fields ### [](#backoff)`backoff` The exponential backoff between retry attempts. The wait starts at `initial_interval` and grows with each failed attempt, up to `max_interval`. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `0s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `3s` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#output)`output` A child output. **Type**: `output` --- # Page 190: salesforce_sink **URL**: https://docs.redpanda.com/connect/components/outputs/salesforce_sink.md --- # salesforce_sink --- title: salesforce_sink latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/salesforce_sink page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/salesforce_sink.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/salesforce_sink.adoc description: Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration. Introduced in version 4.85.0. Consumes batches of messages and writes them to Salesforce. Each message must have a `topic` field (set by the per-topic processor) and a `data` field containing the Salesforce record fields. The `topic` is used to look up the correct `topic_mappings` entry which defines the sObject, operation, and write mode. **Realtime mode** uses the sObject Collections REST API (synchronous, up to 200 records/call). **Bulk mode** uses the Bulk API 2.0 (asynchronous, polls until complete). #### Common ```yml output: label: "" salesforce_sink: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 bulk_batch_size: 1000 max_concurrent_bulk_jobs: 10 bulk_poll_interval: 5s batch_period: 5s max_in_flight: 1 topic_mappings: [] # No default (required) ``` #### Advanced ```yml output: label: "" salesforce_sink: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 bulk_batch_size: 1000 max_concurrent_bulk_jobs: 10 bulk_poll_interval: 5s batch_period: 5s max_in_flight: 1 topic_mappings: [] # No default (required) http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org; check Setup → Company Information. Older versions may lack recent fields. Requires version 4.90.3 or later. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#batch_period)`batch_period` Maximum period to wait before flushing an incomplete batch. **Type**: `string` **Default**: `5s` ### [](#bulk_batch_size)`bulk_batch_size` Number of records per bulk job. Also controls the output batch size. **Type**: `int` **Default**: `1000` ### [](#bulk_poll_interval)`bulk_poll_interval` How often to poll Salesforce for bulk job completion status. **Type**: `string` **Default**: `5s` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive; prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). Requires version 4.90.3 or later. **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#max_concurrent_bulk_jobs)`max_concurrent_bulk_jobs` Maximum number of bulk jobs polling concurrently in the background. Each in-flight job buffers its CSV payload in memory. Lower this value if memory usage is a concern. **Type**: `int` **Default**: `10` ### [](#max_in_flight)`max_in_flight` Maximum number of batches to send concurrently. Increasing this value improves real-time write throughput. **Type**: `int` **Default**: `1` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#topic_mappings)`topic_mappings[]` Per-topic Salesforce write configuration. Each entry maps a topic to an sObject and write settings. **Type**: `array` ### [](#topic_mappings-all_or_none)`topic_mappings[].all_or_none` Real-time only: rolls back every record in a request if any record in it fails. Real-time writes send at most 200 records per request, so a larger batch can be partly committed. **Type**: `bool` **Default**: `false` ### [](#topic_mappings-external_id_field)`topic_mappings[].external_id_field` External ID field name. Required for upsert operations. **Type**: `string` **Default**: `""` ### [](#topic_mappings-mode)`topic_mappings[].mode` Write mode: `realtime` (sObject Collections API) or `bulk` (Bulk API 2.0). **Type**: `string` **Default**: `realtime` ### [](#topic_mappings-operation)`topic_mappings[].operation` Write operation: insert, update, upsert, or delete. **Type**: `string` **Default**: `upsert` ### [](#topic_mappings-sobject)`topic_mappings[].sobject` Salesforce sObject API name (for example, Account, Contact, MyObject\_\_c). **Type**: `string` ### [](#topic_mappings-topic)`topic_mappings[].topic` The topic name to match against the message’s `topic` field. **Type**: `string` --- # Page 191: schema_registry **URL**: https://docs.redpanda.com/connect/components/outputs/schema_registry.md --- # schema_registry --- title: schema_registry latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/schema_registry page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/schema_registry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/schema_registry.adoc description: Publishes schemas to SchemaRegistry. page-git-created-date: "2024-09-04" page-git-modified-date: "2026-10-10" --- Publishes schemas to a schema registry. This output uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). Introduced in version 4.32.2. #### Common ```yml output: label: "" schema_registry: url: "" # No default (required) subject: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml output: label: "" schema_registry: url: "" # No default (required) subject: "" # No default (required) subject_compatibility_level: "" # No default (optional) backfill_dependencies: true translate_ids: false normalize: true remove_metadata: true remove_rule_set: true input_resource: schema_registry_input tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#performance)Performance The `schema_registry` output sends multiple messages in parallel for improved performance. You can use the `max_in_flight` field to tune the maximum number of in-flight messages, or message batches. ## [](#examples)Examples ### [](#write-schemas)Write schemas Write schemas to a Schema Registry instance which is in IMPORT mode, preserving their IDs. Re-writing a schema which already exists is idempotent, so the only errors which need handling are genuine conflicts, where a different schema is already registered under the same ID. These are logged and dropped, while any other error is rejected. ```yaml output: fallback: - schema_registry: url: http://localhost:8082 subject: ${! @schema_registry_subject } subject_compatibility_level: ${! @schema_registry_subject_compatibility_level } - switch: cases: - check: '@fallback_error.contains("Overwrite new schema")' output: drop: {} processors: - log: level: ERROR message: | Subject '${! @schema_registry_subject }' version ${! @schema_registry_version } conflicts with a different schema already registered under ID ${! json("id") }: ${! content() } - output: reject: ${! @fallback_error } ``` ## [](#fields)Fields ### [](#backfill_dependencies)`backfill_dependencies` Backfill missing schema references and previous schema versions. If set to `true`, you must also configure a [`schema_registry`](https://docs.redpanda.com/connect/components/inputs/schema_registry/) input to read source schemas. Requires version 4.40.0 or later. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#input_resource)`input_resource` The label of the [`schema_registry` input](https://docs.redpanda.com/connect/components/inputs/schema_registry/) from which to read source schemas. Requires version 4.40.0 or later. **Type**: `string` **Default**: `schema_registry_input` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#normalize)`normalize` Normalize schemas. Requires version 4.61.0 or later. **Type**: `bool` **Default**: `true` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#remove_metadata)`remove_metadata` Remove the `metadata` property from each schema before it is registered on the destination schema registry. Requires version 4.61.0 or later. **Type**: `bool` **Default**: `true` ### [](#remove_rule_set)`remove_rule_set` Remove the `ruleSet` property from each schema before it is registered on the destination schema registry. Requires version 4.61.0 or later. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` The subject name. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#subject_compatibility_level)`subject_compatibility_level` The compatibility level for the subject. Can be one of `BACKWARD`, `BACKWARD_TRANSITIVE`, `FORWARD`, `FORWARD_TRANSITIVE`, `FULL`, `FULL_TRANSITIVE`, `NONE`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.62.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#translate_ids)`translate_ids` Translate schema IDs. When `false`, schemas are created on the destination with their original IDs and versions, which requires the destination schema registry to be in `IMPORT` mode (Redpanda v25.3 or later, or Confluent Schema Registry). When `true`, the destination assigns new IDs and must be in `READWRITE` mode, since schemas registered without an explicit ID are rejected in `IMPORT` mode. Requires version 4.50.0 or later. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 192: sftp **URL**: https://docs.redpanda.com/connect/components/outputs/sftp.md --- # sftp --- title: sftp latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/sftp page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sftp.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sftp.adoc description: Writes files to an SFTP server. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Writes files to an SFTP server. Introduced in version 3.39.0. In order to have a different path for each object you should use function interpolations described [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ### Common ```yml output: label: "" sftp: address: "" # No default (required) credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" path: "" # No default (required) codec: all-bytes max_in_flight: 64 ``` ### Advanced ```yml output: label: "" sftp: address: "" # No default (required) connection_timeout: 30s credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" ssh_algorithms: additional_key_exchanges: [] # No default (optional) additional_ciphers: [] # No default (optional) additional_macs: [] # No default (optional) path: "" # No default (required) codec: all-bytes max_in_flight: 64 ``` ## [](#fields)Fields ### [](#address)`address` The address (hostname or IP address) of the SFTP server to connect to. **Type**: `string` ### [](#codec)`codec` The way in which the bytes of messages are written to the file. With the `delim:x` codec, each message is followed by the custom delimiter `x`, which can be any character sequence. A delimiter is not added to a message that already ends with it. **Type**: `string` **Default**: `all-bytes` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | delim:x | Append each message to the output stream followed by a custom delimiter. | | lines | Append each message to the output stream followed by a line break. | ```yaml # Examples: codec: lines # --- codec: "delim:\t" # --- codec: delim:foobar ``` ### [](#connection_timeout)`connection_timeout` The connection timeout to use when connecting to the target server. Requires version 4.59.0 or later. **Type**: `string` **Default**: `30s` ### [](#credentials)`credentials` The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access. **Type**: `object` ### [](#credentials-host_public_key)`credentials.host_public_key` The raw contents of the SFTP server’s public key, used for host key verification. Requires version 4.59.0 or later. **Type**: `string` ### [](#credentials-host_public_key_file)`credentials.host_public_key_file` The path to the SFTP server’s public key file, used for host key verification. Requires version 4.59.0 or later. **Type**: `string` ### [](#credentials-password)`credentials.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-private_key)`credentials.private_key` The raw contents of the private key used to authenticate with the SFTP server. This field provides an alternative to `private_key_file`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.51.0 or later. **Type**: `string` ### [](#credentials-private_key_file)`credentials.private_key_file` The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the `private_key` field. **Type**: `string` ### [](#credentials-private_key_pass)`credentials.private_key_pass` An optional passphrase for decrypting the private key, if it is encrypted. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-ssh_algorithms)`credentials.ssh_algorithms` Additional SSH transport algorithms to permit for compatibility with legacy SFTP servers. Configured algorithms are appended to the default algorithms, which remain preferred, and when unset the defaults are used unchanged. Algorithms the underlying SSH library classifies as insecure weaken transport security and should only be enabled when required by a known server. These settings do not affect host key verification. Requires version 4.113.0 or later. **Type**: `object` ### [](#credentials-ssh_algorithms-additional_ciphers)`credentials.ssh_algorithms.additional_ciphers[]` Ciphers to permit in addition to the defaults, for example `aes128-cbc`. **Type**: `array` ### [](#credentials-ssh_algorithms-additional_key_exchanges)`credentials.ssh_algorithms.additional_key_exchanges[]` Key exchange algorithms to permit in addition to the defaults, for example `diffie-hellman-group1-sha1` or `diffie-hellman-group-exchange-sha1`. **Type**: `array` ### [](#credentials-ssh_algorithms-additional_macs)`credentials.ssh_algorithms.additional_macs[]` MAC algorithms to permit in addition to the defaults. **Type**: `array` ### [](#credentials-username)`credentials.username` The username required to authenticate with the SFTP server. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The file to save the messages to on the SFTP server. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 193: slack_post **URL**: https://docs.redpanda.com/connect/components/outputs/slack_post.md --- # slack_post --- title: slack_post latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/slack_post page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/slack_post.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/slack_post.adoc description: Posts a new message to a Slack channel using the Slack API method chat.postMessage. page-git-created-date: "2025-05-02" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Posts a new message to a Slack channel using the Slack API method [chat.postMessage](https://api.slack.com/methods/chat.postMessage). Introduced in version 4.52.0. #### Common ```yml output: label: "" slack_post: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" text: "" blocks: "" # No default (optional) markdown: true unfurl_links: false unfurl_media: true link_names: false ``` #### Advanced ```yml output: label: "" slack_post: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" text: "" blocks: "" # No default (optional) markdown: true unfurl_links: false unfurl_media: true link_names: false ``` See also: [Examples](#examples) ## [](#fields)Fields ### [](#blocks)`blocks` A Bloblang query that should return a JSON array of [Slack blocks](https://api.slack.com/reference/block-kit/blocks). You can either specify message content in the `text` or `blocks` fields, but not both. **Type**: `string` ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the correct permissions to post messages to the target Slack channel. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the target Slack channel. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#link_names)`link_names` When set to `true`, this output finds and links to [user groups](https://api.slack.com/reference/surfaces/formatting#mentioning-groups) mentioned in Slack messages. **Type**: `bool` **Default**: `false` ### [](#markdown)`markdown` When set to `true`, this output accepts message content in Markdown format and applies Markdown formatting to the message. **Type**: `bool` **Default**: `true` ### [](#text)`text` The text content of the message. You can either specify message content in the `text` or `blocks` fields, but not both. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#thread_ts)`thread_ts` Specify the thread timestamp (`ts` value) of another message to post a reply within the same thread. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#unfurl_links)`unfurl_links` When set to `true`, this output provides previews of linked content in Slack messages. For more information about unfurling links, see the [Slack documentation](https://api.slack.com/reference/messaging/link-unfurling). **Type**: `bool` **Default**: `false` ### [](#unfurl_media)`unfurl_media` When set to `true`, this output enables media unfurling, which provides previews of rich content in Slack messages, such as videos or embedded tweets. **Type**: `bool` **Default**: `true` ## [](#examples)Examples ### [](#echo-slackbot)Echo Slackbot A slackbot that echo messages from other users ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` --- # Page 194: slack_reaction **URL**: https://docs.redpanda.com/connect/components/outputs/slack_reaction.md --- # slack_reaction --- title: slack_reaction latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/slack_reaction page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/slack_reaction.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/slack_reaction.adoc description: Add or remove an emoji reaction to a Slack message. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Add or remove an emoji reaction to a Slack message using [`reactions.add`](https://api.slack.com/methods/reactions.add) and [`reactions.remove`](https://api.slack.com/methods/reactions.remove). #### Common ```yml output: label: "" slack_reaction: bot_token: "" # No default (required) channel_id: "" # No default (required) timestamp: "" # No default (required) emoji: "" # No default (required) action: add max_in_flight: 64 ``` #### Advanced ```yml output: label: "" slack_reaction: bot_token: "" # No default (required) channel_id: "" # No default (required) timestamp: "" # No default (required) emoji: "" # No default (required) action: add max_in_flight: 64 ``` > 📝 **NOTE** > > Raising `max_in_flight` can improve throughput in high-volume pipelines, but it can also exceed Slack API rate limits. ## [](#fields)Fields ### [](#action)`action` Whether to add or remove the reaction. When set to `add`, the specified emoji reaction is applied to the target message. When set to `remove`, the emoji reaction is removed from the target message. **Type**: `string` **Default**: `add` **Options**: `add`, `remove` ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the [`reactions:write` scope](https://api.slack.com/scopes/reactions:write). The token starts with `xoxb-`. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the Slack channel that contains the target message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#emoji)`emoji` The name of the emoji to be added or removed, without surrounding colons. Use the plain emoji name, such as `thumbsup` or `tada`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#timestamp)`timestamp` The timestamp of the message to react to. This is a unique identifier for the message, usually obtained from a previous Slack API call (such as `chat.postMessage` or `conversations.history`). It typically looks like a Unix timestamp with a decimal. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 195: snowflake_put **URL**: https://docs.redpanda.com/connect/components/outputs/snowflake_put.md --- # snowflake_put --- title: snowflake_put latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/snowflake_put page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/snowflake_put.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/snowflake_put.adoc description: Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > 💡 **TIP** > > Use the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) for improved performance, cost-effectiveness, and ease of use. Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables. Introduced in version 4.0.0. #### Common ```yml output: label: "" snowflake_put: account: "" # No default (required) region: "" # No default (optional) cloud: "" # No default (optional) user: "" # No default (required) password: "" # No default (optional) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) role: "" # No default (required) database: "" # No default (required) warehouse: "" # No default (required) schema: "" # No default (required) stage: "" # No default (required) path: "" file_name: "" file_extension: "" compression: AUTO request_id: "" snowpipe: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" max_in_flight: 1 ``` #### Advanced ```yml output: label: "" snowflake_put: account: "" # No default (required) region: "" # No default (optional) cloud: "" # No default (optional) user: "" # No default (required) password: "" # No default (optional) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) role: "" # No default (required) database: "" # No default (required) warehouse: "" # No default (required) schema: "" # No default (required) stage: "" # No default (required) path: "" file_name: "" file_extension: "" upload_parallel_threads: 4 compression: AUTO request_id: "" snowpipe: "" # No default (optional) client_session_keep_alive: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 1 ``` In order to use a different stage and / or Snowpipe for each message, you can use function interpolations as described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When using batching, messages are grouped by the calculated stage and Snowpipe and are streamed to individual files in their corresponding stage and, optionally, a Snowpipe `insertFiles` REST API call will be made for each individual file. ## [](#credentials)Credentials Two authentication mechanisms are supported: - User/password - Key Pair Authentication ### [](#userpassword)User/password This is a basic authentication mechanism which allows you to PUT data into a stage. However, it is not compatible with Snowpipe. ### [](#key-pair-authentication)Key pair authentication This authentication mechanism allows Snowpipe functionality, but it does require configuring an SSH Private Key beforehand. Please consult the [documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication) for details on how to set it up and assign the Public Key to your user. Note that the Snowflake documentation [used to suggest](https://twitter.com/felipehoffa/status/1560811785606684672) using this command: ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 ``` to generate an encrypted SSH private key. However, in this case, it uses an encryption algorithm called `pbeWithMD5AndDES-CBC`, which is part of the PKCS#5 v1.5 and is considered insecure. Due to this, Redpanda Connect does not support it and, if you wish to use password-protected keys directly, you must use PKCS#5 v2.0 to encrypt them by using the following command (as the current Snowflake docs suggest): ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8 ``` If you have an existing key encrypted with PKCS#5 v1.5, you can re-encrypt it with PKCS#5 v2.0 using this command: ```bash openssl pkcs8 -in rsa_key_original.p8 -topk8 -v2 des3 -out rsa_key.p8 ``` Please consult the [pkcs8 command documentation](https://linux.die.net/man/1/pkcs8) for details on PKCS#5 algorithms. ## [](#batching)Batching It’s common to want to upload messages to Snowflake as batched archives. The easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](https://docs.redpanda.com/connect/components/processors/archive/) and/or [`compress`](https://docs.redpanda.com/connect/components/processors/compress/) processor. For the optimal batch size, please consult the Snowflake [documentation](https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare.html). ## [](#snowpipe)Snowpipe Given a table called `BENTHOS_TBL` with one column of type `variant`: ```sql CREATE OR REPLACE TABLE BENTHOS_DB.PUBLIC.BENTHOS_TBL(RECORD variant) ``` and the following `BENTHOS_PIPE` Snowpipe: ```sql CREATE OR REPLACE PIPE BENTHOS_DB.PUBLIC.BENTHOS_PIPE AUTO_INGEST = FALSE AS COPY INTO BENTHOS_DB.PUBLIC.BENTHOS_TBL FROM (SELECT * FROM @%BENTHOS_TBL) FILE_FORMAT = (TYPE = JSON COMPRESSION = AUTO) ``` you can configure Redpanda Connect to use the implicit table stage `@%BENTHOS_TBL` as the `stage` and `BENTHOS_PIPE` as the `snowpipe`. In this case, you must set `compression` to `AUTO` and, if using message batching, you’ll need to configure an [`archive`](https://docs.redpanda.com/connect/components/processors/archive/) processor with the `concatenate` format. Since the `compression` is set to `AUTO`, the [gosnowflake](https://github.com/snowflakedb/gosnowflake) client library will compress the messages automatically so you don’t need to add a [`compress`](https://docs.redpanda.com/connect/components/processors/compress/) processor for message batches. If you add `STRIP_OUTER_ARRAY = TRUE` in your Snowpipe `FILE_FORMAT` definition, then you must use `json_array` instead of `concatenate` as the archive processor format. > 📝 **NOTE** > > Only Snowpipes with `FILE_FORMAT` `TYPE` `JSON` are currently supported. ## [](#snowpipe-troubleshooting)Snowpipe troubleshooting Snowpipe [provides](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html) the `insertReport` and `loadHistoryScan` REST API endpoints which can be used to get information about recent Snowpipe calls. In order to query them, you’ll first need to generate a valid JWT token for your Snowflake account. There are two methods for doing so: - Using the `snowsql` [utility](https://docs.snowflake.com/en/user-guide/snowsql.html): ```bash snowsql --private-key-path rsa_key.p8 --generate-jwt -a -u ``` - Using the Python `sql-api-generate-jwt` [utility](https://docs.snowflake.com/en/developer-guide/sql-api/authenticating.html#generating-a-jwt-in-python): ```bash python3 sql-api-generate-jwt.py --private_key_file_path=rsa_key.p8 --account= --user= ``` Once you successfully generate a JWT token and store it into the `JWT_TOKEN` environment variable, then you can, for example, query the `insertReport` endpoint using `curl`: ```bash curl -H "Authorization: Bearer ${JWT_TOKEN}" "https://.snowflakecomputing.com/v1/data/pipes/../insertReport" ``` If you need to pass in a valid `requestId` to any of these Snowpipe REST API endpoints, you can set a [uuid\_v4()](https://docs.redpanda.com/connect/guides/bloblang/functions/#uuid_v4) string in a metadata field called `request_id`, log it via the [`log`](https://docs.redpanda.com/connect/components/processors/log/) processor and then configure `request_id: ${ @request_id }` ). Alternatively, you can [enable debug logging](https://docs.redpanda.com/connect/components/logger/about/) and Redpanda Connect will print the Request IDs that it sends to Snowpipe. ## [](#general-troubleshooting)General troubleshooting The underlying [`gosnowflake` driver](https://github.com/snowflakedb/gosnowflake) requires write access to the default directory to use for temporary files. Please consult the [`os.TempDir`](https://pkg.go.dev/os#TempDir) docs for details on how to change this directory via environment variables. A silent failure can occur due to [this issue](https://github.com/snowflakedb/gosnowflake/issues/701), where the underlying [`gosnowflake` driver](https://github.com/snowflakedb/gosnowflake) doesn’t return an error and doesn’t log a failure if it can’t figure out the current username. One way to trigger this behavior is by running Redpanda Connect in a Docker container with a non-existent user ID (such as `--user 1000:1000`). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ## [](#examples)Examples ### [](#kafka-realtime-brokers)Kafka / realtime brokers Upload message batches from realtime brokers such as Kafka persisting the batch partition and offsets in the stage path and filename similarly to the [Kafka Connector scheme](https://docs.snowflake.com/en/user-guide/kafka-connector-ts.html#step-1-view-the-copy-history-for-the-table) and call Snowpipe to load them into a table. When batching is configured at the input level, it is done per-partition. ```yaml input: redpanda: seed_brokers: - localhost:9092 topics: - foo consumer_group: rpcn max_yield_batch_bytes: 8MB processors: - mapping: | meta kafka_start_offset = meta("kafka_offset").from(0) meta kafka_end_offset = meta("kafka_offset").from(-1) meta batch_timestamp = if batch_index() == 0 { now() } - mapping: | meta batch_timestamp = if batch_index() != 0 { meta("batch_timestamp").from(0) } output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos/BENTHOS_TBL/${! @kafka_partition } file_name: ${! @kafka_start_offset }_${! @kafka_end_offset }_${! meta("batch_timestamp") } upload_parallel_threads: 4 compression: NONE snowpipe: BENTHOS_PIPE ``` ### [](#no-compression)No compression Upload concatenated messages into a `.json` file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: NONE batching: count: 10 period: 3s processors: - archive: format: concatenate ``` ### [](#parquet-format-with-snappy-compression)Parquet format with snappy compression Upload concatenated messages into a `.parquet` file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos file_extension: parquet upload_parallel_threads: 4 compression: NONE batching: count: 10 period: 3s processors: - parquet_encode: schema: - name: ID type: INT64 - name: CONTENT type: BYTE_ARRAY default_compression: snappy ``` ### [](#automatic-compression)Automatic compression Upload concatenated messages compressed automatically into a `.gz` archive file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: AUTO batching: count: 10 period: 3s processors: - archive: format: concatenate ``` ### [](#deflate-compression)DEFLATE compression Upload concatenated messages compressed into a `.deflate` archive file to a table stage and call Snowpipe to load them into a table. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: DEFLATE snowpipe: BENTHOS_PIPE batching: count: 10 period: 3s processors: - archive: format: concatenate - mapping: | root = content().compress("zlib") ``` ### [](#raw_deflate-compression)RAW_DEFLATE compression Upload concatenated messages compressed into a `.raw_deflate` archive file to a table stage and call Snowpipe to load them into a table. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: RAW_DEFLATE snowpipe: BENTHOS_PIPE batching: count: 10 period: 3s processors: - archive: format: concatenate - mapping: | root = content().compress("flate") ``` ## [](#fields)Fields ### [](#account)`account` Account name, which is the same as the [Account Identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#where-are-account-identifiers-used). However, when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier), the Account Identifier is formatted as `..` and this field needs to be populated using the `` part. **Type**: `string` ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_session_keep_alive)`client_session_keep_alive` Enable Snowflake keepalive mechanism to prevent the client session from expiring after 4 hours (error 390114). Requires version 4.11.0 or later. **Type**: `bool` **Default**: `false` ### [](#cloud)`cloud` Optional cloud platform field which needs to be populated when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier) and it must be set to the `` part of the Account Identifier (`..`). **Type**: `string` ```yaml # Examples: cloud: aws # --- cloud: gcp # --- cloud: azure ``` ### [](#compression)`compression` Compression type. **Type**: `string` **Default**: `AUTO` | Option | Summary | | --- | --- | | AUTO | Compression (gzip) is applied automatically by the output and messages must contain plain-text JSON. Default file_extension: gz. | | DEFLATE | Messages must be pre-compressed using the zlib algorithm (with zlib header, RFC1950). Default file_extension: deflate. | | GZIP | Messages must be pre-compressed using the gzip algorithm. Default file_extension: gz. | | NONE | No compression is applied and messages must contain plain-text JSON. Default file_extension: json. | | RAW_DEFLATE | Messages must be pre-compressed using the flate algorithm (without header, RFC1951). Default file_extension: raw_deflate. | | ZSTD | Messages must be pre-compressed using the Zstandard algorithm. Default file_extension: zst. | ### [](#database)`database` Database. **Type**: `string` ### [](#file_extension)`file_extension` Stage file extension. Will be derived from the configured `compression` if not set or empty. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: file_extension: csv # --- file_extension: parquet ``` ### [](#file_name)`file_name` Stage file name. Will be equal to the Request ID if not set or empty. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at any given time. Batches share one Snowflake connection and are uploaded one at a time, so a higher value lets batches queue but doesn’t upload them in parallel. To upload the files of a batch in parallel, use `upload_parallel_threads`. **Type**: `int` **Default**: `1` ### [](#password)`password` An optional password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#path)`path` Stage path. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#private_key)`private_key` Your private SSH key. When using encrypted keys, you must also set a value for `private_key_pass`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ### [](#private_key_file)`private_key_file` The path to a file containing your private SSH key. When using encrypted keys, you must also set a value for `private_key_pass`. **Type**: `string` ### [](#private_key_pass)`private_key_pass` The passphrase for your private SSH key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#region)`region` Optional region field which needs to be populated when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier) and it must be set to the `` part of the Account Identifier (`..`). **Type**: `string` ```yaml # Examples: region: us-west-2 ``` ### [](#request_id)`request_id` Request ID. Will be assigned a random UUID (v4) string if not set or empty. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` ### [](#role)`role` Role. **Type**: `string` ### [](#schema)`schema` Schema. **Type**: `string` ### [](#snowpipe-2)`snowpipe` An optional Snowpipe name. Use the `` part from `..`. `private_key` or `private_key_file` must be set when using this feature. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#stage)`stage` Stage name. Use either one of the [supported](https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage.html) stage types. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#upload_parallel_threads)`upload_parallel_threads` Specifies the number of threads to use for uploading files. **Type**: `int` **Default**: `4` ### [](#user)`user` Username. **Type**: `string` ### [](#warehouse)`warehouse` Warehouse. **Type**: `string` --- # Page 196: snowflake_streaming **URL**: https://docs.redpanda.com/connect/components/outputs/snowflake_streaming.md --- # snowflake_streaming --- title: snowflake_streaming latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/snowflake_streaming page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/snowflake_streaming.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/snowflake_streaming.adoc description: Ingest data into Snowflake using the Snowpipe Streaming classic architecture. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Ingest data from your pipeline into Snowflake using the [Snowpipe Streaming classic architecture](https://docs.snowflake.com/en/user-guide/snowpipe-streaming/snowpipe-streaming-classic-overview). This output does not work with the Snowpipe Streaming high-performance architecture. To help you configure your own `snowflake_streaming` output, this page includes [example data pipelines](#examples). Introduced in version 4.39.0. #### Common ```yml output: label: "" snowflake_streaming: account: "" # No default (required) user: "" # No default (required) role: "" # No default (required) database: "" # No default (required) schema: "" # No default (required) table: "" # No default (required) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) mapping: "" # No default (optional) init_statement: "" # No default (optional) schema_evolution: enabled: false # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" max_in_flight: 4 ``` #### Advanced ```yml output: label: "" snowflake_streaming: account: "" # No default (required) url: "" # No default (optional) user: "" # No default (required) role: "" # No default (required) database: "" # No default (required) schema: "" # No default (required) table: "" # No default (required) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) mapping: "" # No default (optional) init_statement: "" # No default (optional) schema_evolution: enabled: false # No default (required) ignore_nulls: true processors: [] # No default (optional) build_options: parallelism: 1 chunk_size: 50000 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 channel_prefix: "" # No default (optional) channel_name: "" # No default (optional) offset_token: "" # No default (optional) commit_backoff: initial_interval: 32ms max_interval: 512ms max_elapsed_time: 60s multiplier: 2 message_format: object timestamp_format: 2006-01-02T15:04:05.999999999Z07:00 ``` ## [](#conversion-of-message-data-into-snowflake-table-rows)Conversion of message data into Snowflake table rows Message data conversion to Snowflake table rows is determined by the: - Output message contents. - [Schema evolution settings](#schema_evolution). - Schema of the [target Snowflake table](#table). The following scenarios highlight how these three factors affect data written to the target table. > 📝 **NOTE** > > For reduced complexity, consider [turning on schema evolution](#schema_evolution), which automatically creates and updates the Snowflake table schema based on message contents. ### [](#scenario-data-and-table-schema-match-schema-evolution-turned-on-or-off)Scenario: Data and table schema match (schema evolution turned on or off) An output message matches the existing table schema, and the `schema_evolution.enabled` field is set to `true` or `false`. The target Snowflake table has two columns: - `product_id` (NUMBER) - `product_code` (STRING) A pipeline generates the following message: ```json {"product_id": 521, "product_code": “EST-PR”} ``` In this scenario: - The JSON keys in the message (`"product_id"` and `"product_code"`) match column names in the target Snowflake table. - The message values match the column data types. (If there was a data mismatch, the message would be rejected.) - Redpanda Connect inserts the message values into a new row in the target Snowflake table. | product_id | product_code | | --- | --- | | 521 | EST-PR | ### [](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-on)Scenario: Data and table schema mismatch (schema evolution turned on) An output message includes schema updates, and the `schema_evolution.enabled` field is set to `true`. The target Snowflake table has the same two columns as the [previous scenario](#scenario-data-and-table-schema-match-schema-evolution-turned-on-or-off): - `product_id` (NUMBER) - `product_code` (STRING) This time, the pipeline generates the following message: ```json {"product_batch": 11111, "product_color": “yellow”} ``` In this scenario: - The JSON keys (`"product_batch"` and `"product_color"`) do not match column names in the target Snowflake table. - As schema evolution is enabled, Redpanda Connect adds two new columns to the target table with data types derived from the output message values. For more information about the mapping of data types, see [Supported data formats for Snowflake columns](#supported-data-formats-for-snowflake-columns). - Redpanda Connect inserts the message values into a new table row. | product_id | product_code | product_batch | product_color | | --- | --- | --- | --- | | (null) | (null) | 11111 | yellow | > 📝 **NOTE** > > You can [configure processors](#schema_evolution-processors) to override the schema updates derived from the message values. ### [](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-off)Scenario: Data and table schema mismatch (schema evolution turned off) An output message includes schema updates, and the `schema_evolution.enabled` field is set to `false`. The target Snowflake table has the same two columns: - `product_id` (NUMBER) - `product_code` (STRING) The pipeline generates the same message as the [previous scenario](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-on): ```json {"product_batch": 11111, "product_color": “yellow”} ``` In this scenario: - The JSON keys (`"product_batch"` and `"product_color"`) do not match any existing column names. - Because schema evolution is turned off, Redpanda Connect ignores the extra column names and values and inserts a row of null values. | product_id | product_code | | --- | --- | | (null) | (null) | ## [](#supported-data-formats-for-snowflake-columns)Supported data formats for Snowflake columns The message data from your output must match the columns in the Snowflake table that you want to write data to. The following table shows you the [column data types supported by Snowflake](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types) and how they correspond to the [Bloblang data types](https://docs.redpanda.com/connect/guides/bloblang/methods/#type) in Redpanda Connect. | Snowflake column data type | Bloblang data types | | --- | --- | | CHAR, VARCHAR | string | | BINARY | string or bytes | | NUMBER | number, or string where the string is parsed into a number | | FLOAT, including special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity) | number | | BOOLEAN | bool, or number where a non-zero number is true | | TIME, DATE, TIMESTAMP | timestamp, or number where the number is a converted to a Unix timestamp, or string where the string is parsed using RFC 3339 format | | VARIANT, ARRAY, OBJECT | Any data type converted into JSON | | GEOGRAPHY,GEOMETRY | Not supported | ## [](#authentication)Authentication You can authenticate with Snowflake using an [RSA key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth). Either specify: - A PEM-encoded private key, in the [`private_key` field](#private_key). - The path to a file from which the output can load the private RSA key, in the [`private_key_file` field](#private_key_file). ## [](#performance)Performance For improved performance, this output: - Sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches) with the field `max_in_flight`. - Sends messages as a batch. You can configure batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ### [](#batch-sizes)Batch sizes Redpanda recommends that every message batch writes at least 16 MiB of compressed output to Snowflake. You can monitor batch sizes using the `snowflake_compressed_output_size_bytes` metric. ### [](#metrics)Metrics This output emits the following metrics. | Metric name | Description | | --- | --- | | snowflake_compressed_output_size_bytes | The size in bytes of each message batch uploaded to Snowflake. | | snowflake_convert_latency_ns | The time taken to convert messages into the Snowflake column data types. | | snowflake_serialize_latency_ns | The time taken to serialize the converted columnar data into a file for upload to Snowflake. | | snowflake_build_output_latency_ns | The time taken to build the file that is uploaded to Snowflake. This metric is the sum of snowflake_convert_latency_ns + snowflake_serialize_latency_ns. | | snowflake_upload_latency_ns | The time taken to upload the output file to Snowflake. | | snowflake_register_latency_ns | The time taken to register the uploaded output file with Snowflake. | | snowflake_commit_latency_ns | The time taken to commit the uploaded data updates to the target Snowflake table. | ## [](#fields)Fields ### [](#account)`account` The [Snowflake account name to use](https://docs.snowflake.com/en/user-guide/admin-account-identifier#account-name). Use the format `-` where: - The `` is the name of your Snowflake organization. - The `` is the unique name of your account within your Snowflake organization. To find the correct value for this field, run the following query in Snowflake: ```sql WITH HOSTLIST AS (SELECT * FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$allowlist())))) SELECT REPLACE(VALUE:host,'.snowflakecomputing.com','') AS ACCOUNT_IDENTIFIER FROM HOSTLIST WHERE VALUE:type = 'SNOWFLAKE_DEPLOYMENT_REGIONLESS'; ``` **Type**: `string` ```yaml # Examples: account: ORG-ACCOUNT ``` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#build_options)`build_options` Options for optimizing the build of the output data that is sent to Snowflake. Monitor the `snowflake_build_output_latency_ns` metric to assess whether you need to update these options. Requires version 4.40.0 or later. **Type**: `object` ### [](#build_options-chunk_size)`build_options.chunk_size` The number of table rows to submit in each chunk for processing. **Type**: `int` **Default**: `50000` ### [](#build_options-parallelism)`build_options.parallelism` The maximum amount of parallel processing to use when building the output for Snowflake. **Type**: `int` **Default**: `1` ### [](#channel_name)`channel_name` The channel name to use when connecting to a Snowflake table. Duplicate channel names cause errors and prevent multiple instances of Redpanda Connect from writing at the same time. Redpanda Connect assumes that a message batch contains messages for a single channel, which means that interpolation is only executed on the first message in each batch. If your pipeline uses an input that is partitioned, such as an Apache Kafka topic, batch messages at the input level to make sure all messages in a batch are written to the same channel. You can specify either the `channel_name` or `channel_prefix`, but not both. If neither field is populated, this output creates a channel name based on a table’s fully-qualified name, which results in a single stream per table. > 📝 **NOTE** > > Snowflake limits the number of streams per table to 10,000. If you need to use more than 10,000 streams, contact [Snowflake support](https://www.snowflake.com/en/support/). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.45.0 or later. **Type**: `string` ```yaml # Examples: channel_name: partition-${!@kafka_partition} ``` ### [](#channel_prefix)`channel_prefix` The prefix to use when creating a channel name for connecting to a Snowflake table. Adding a `channel_prefix` avoids the creation of duplicate channel names, which result in errors and prevent multiple instances of Redpanda Connect from writing at the same time. You can specify either the `channel_prefix` or `channel_name`, but not both. If neither field is populated, this output creates a channel name based on a table’s fully-qualified name, which results in a single stream per table. The maximum number of channels open at any time is determined by the value in the `max_in_flight` field. > 📝 **NOTE** > > Snowflake limits the number of streams per table to 10,000. If you need to use more than 10,000 streams, contact [Snowflake support](https://www.snowflake.com/en/support/). **Type**: `string` ```yaml # Examples: channel_prefix: channel-${HOST} ``` ### [](#commit_backoff)`commit_backoff` Control how frequently Snowflake is polled to check if data has been committed. Requires version 4.82.0 or later. **Type**: `object` ### [](#commit_backoff-initial_interval)`commit_backoff.initial_interval` The initial period to wait between status polls. **Type**: `string` **Default**: `32ms` ### [](#commit_backoff-max_elapsed_time)`commit_backoff.max_elapsed_time` The maximum total time to wait for data to be committed. If zero then no limit is used. **Type**: `string` **Default**: `60s` ### [](#commit_backoff-max_interval)`commit_backoff.max_interval` The maximum period to wait between status polls. **Type**: `string` **Default**: `512ms` ### [](#commit_backoff-multiplier)`commit_backoff.multiplier` The factor by which the poll interval grows on each attempt. **Type**: `float` **Default**: `2` ### [](#database)`database` The Snowflake database you want to write data to. **Type**: `string` ```yaml # Examples: database: MY_DATABASE ``` ### [](#init_statement)`init_statement` Optional SQL statements to execute immediately after this output connects to Snowflake for the first time. This is a useful way to initialize tables before processing data. > 📝 **NOTE** > > Make sure your SQL statements are idempotent, so they do not cause issues when run multiple times after service restarts. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS mytable (amount NUMBER); # --- init_statement: | ALTER TABLE t1 ALTER COLUMN c1 DROP NOT NULL; ALTER TABLE t1 ADD COLUMN a2 NUMBER; ``` ### [](#mapping)`mapping` The [Bloblang `mapping`](https://docs.redpanda.com/connect/guides/bloblang/about/) to execute on each message. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `4` ### [](#message_format)`message_format` The format to expect incoming messages from the rest of the pipeline. Requires version 4.79.0 or later. **Type**: `string` **Default**: `object` | Option | Summary | | --- | --- | | array | Messages are an array of values where the position in the array matches up the with ordinal of the column in snowflake | | object | Messages are an object in JSON or bloblang where the key of the object is the column name in snowflake and the value is the value for the column | ```yaml # Examples: message_format: array ``` ### [](#offset_token)`offset_token` The offset token to use for exactly-once delivery of data to a Snowflake table. This output assumes that messages within a batch are in increasing order by offset token. When data is sent on a channel, the offset token of each message in the batch is compared to the latest token processed by the channel. If the offset token is less than the latest token, it’s assumed the message is a duplicate and is dropped. Messages must be delivered to the output in order, otherwise they are processed as duplicates and dropped. Retried messages can also be seen as duplicates if later messages have succeeded in the meantime, so in most cases use a dead-letter queue to process failed messages. See the `Ingesting data exactly once from Redpanda` example. If both offset tokens parse as base-10 integers (up to 64 bits), they are compared numerically, so a bare numeric token such as `${!@kafka_offset}` does not need padding. Any other token, including numbers outside of the 64-bit integer range or a numeric value combined with a prefix or separator, falls back to a lexicographic comparison of its string representation. If you use one of these tokens, pad it so that it’s lexicographically ordered in its string representation. For more information about offset tokens, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.45.0 or later. **Type**: `string` ```yaml # Examples: offset_token: offset-${!"%016X".format(@kafka_offset)} # --- offset_token: postgres-${!@lsn} ``` ### [](#private_key)`private_key` The PEM-encoded private RSA key to use for authentication with Snowflake. You must specify a value for this field or the `private_key_file` field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#private_key_file)`private_key_file` A `.p8`, PEM-encoded file to load the private RSA key from. You must specify a value for this field or the `private_key` field. **Type**: `string` ### [](#private_key_pass)`private_key_pass` If the RSA key is encrypted, specify the RSA key passphrase. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#role)`role` The role of the user specified in the `user` field. The user’s role must have the [required privileges](https://docs.snowflake.com/en/user-guide/snowpipe-streaming/snowpipe-streaming-classic-configuration#authentication-and-authorization) to call the Snowpipe Streaming APIs. For more information about user roles, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/admin-user-management#user-roles). **Type**: `string` ```yaml # Examples: role: ACCOUNTADMIN ``` ### [](#schema)`schema` The schema of the Snowflake database you want to write data to. **Type**: `string` ```yaml # Examples: schema: PUBLIC ``` ### [](#schema_evolution)`schema_evolution` Options to control schema updates when messages are written to the Snowflake table, such as adding columns as new fields appear in messages. **Type**: `object` ### [](#schema_evolution-enabled)`schema_evolution.enabled` Whether schema evolution is enabled. When set to `true`, the Snowflake table is automatically created based on the schema of the first message written to it, if the table does not already exist. As new fields are added to subsequent messages in the pipeline, new columns are created in the Snowflake table. Any required columns are marked as `nullable` if new messages do not include data for them. **Type**: `bool` ### [](#schema_evolution-ignore_nulls)`schema_evolution.ignore_nulls` When set to `true` and schema evolution is enabled, new columns that have `null` values _are not_ added to the Snowflake table and do not trigger schema evolution. When set to `false`, `null` columns trigger schema migrations in Snowflake. Ignoring `null` values: - Prevents unnecessary schema changes caused by placeholder or incomplete data. - Avoids creating table columns with incorrect data types. > 📝 **NOTE** > > Redpanda does not recommend changing the default setting (`true`) unless you know the data type of `null` columns in advance. Requires version 4.48.0 or later. **Type**: `bool` **Default**: `true` ### [](#schema_evolution-processors)`schema_evolution.processors[]` A series of processors to execute when new columns are added to the Snowflake table. You can use these processors to: - Run side effects when the schema evolves. - Enrich the message with additional information to guide the schema changes. For example, a processor could read the schema from the schema registry that a message was produced with and use that schema to determine the data type of the new column in Snowflake. The input to these processors is an object with the value and name of the new column, the original message, and details of the Snowflake table the output writes to. The metadata remains the same as in the original message that triggered the schema update. For example: `{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "foo"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}` The output from the processors must be a single message that contains a string specifying the column data type to use, such as `FLOAT`, `VARIANT`, or `NUMBER(38, 0)`. The output then runs an `ALTER TABLE` statement on the Snowflake table to add the column with the corresponding data type. Requires version 4.46.0 or later. **Type**: `array` ```yaml # Examples: processors: - mapping: |- root = match this.value.type() { this == "string" => "STRING" this == "bytes" => "BINARY" this == "number" => "DOUBLE" this == "bool" => "BOOLEAN" this == "timestamp" => "TIMESTAMP" _ => "VARIANT" } ``` ### [](#table)`table` The Snowflake table you want to write data to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table: MY_TABLE ``` ### [](#timestamp_format)`timestamp_format` The format to parse string values for `TIMESTAMP`, `TIMESTAMP_LTZ` and `TIMESTAMP_NTZ` columns. Should be a layout for [time.Parse](https://pkg.go.dev/time#Parse) in Go. Requires version 4.79.0 or later. **Type**: `string` **Default**: `2006-01-02T15:04:05.999999999Z07:00` ### [](#url)`url` Specify a custom URL to connect to Snowflake. This parameter overrides the default URL, which is generated from the value of the `account` field: `[https://.snowflakecomputing.com](https://\.snowflakecomputing.com)`. Requires version 4.48.0 or later. **Type**: `string` ```yaml # Examples: url: https://org-account.privatelink.snowflakecomputing.com ``` ### [](#user)`user` Specify a user to run the Snowpipe Stream. To learn how to create a user, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/admin-user-management). **Type**: `string` ## [](#examples)Examples ### [](#exactly-once-cdc-into-snowflake)Exactly once CDC into Snowflake How to send data from a PostgreSQL table into Snowflake exactly once using Postgres Logical Replication. > 📝 **NOTE** > > If attempting to do exactly-once it’s important that rows are delivered in order to the output. Be sure to read the documentation for offset\_token first. Removing the offset\_token is a safer option that will instruct Redpanda Connect to use its default at-least-once delivery model instead. ```yaml input: postgres_cdc: dsn: postgres://foouser:foopass@localhost:5432/foodb schema: "public" slot_name: "my_repl_slot" tables: ["my_pg_table"] # We want very large batches - each batch will be sent to Snowflake individually # so to optimize query performance we want as big of files as we have memory for batching: count: 50000 period: 45s # Prevent multiple batches from being in flight at once, so that we never send # a batch while another batch is being retried, this is important to ensure that # the Snowflake Snowpipe Streaming channel does not see older data - as it will # assume that the older data is already committed. checkpoint_limit: 1 output: snowflake_streaming: # We use the log sequence number in the WAL from Postgres to ensure we # only upload data exactly once, these are already lexicographically # ordered. offset_token: "${!@lsn}" # Since we're sending a single ordered log, we can only send one thing # at a time to ensure that we're properly incrementing our offset_token # and only using a single channel at a time. max_in_flight: 1 account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MY_PG_TABLE" private_key_file: "my/private/key.p8" ``` ### [](#ingesting-data-exactly-once-from-redpanda)Ingesting data exactly once from Redpanda How to ingest data from Redpanda with consumer groups, decode the schema using the schema registry, then write the corresponding data into Snowflake exactly once. > 📝 **NOTE** > > If attempting to do exactly-once it’s important that records are delivered in order to the output and correctly partitioned. Be sure to read the documentation for `channel_name` and `offset_token` first. Removing the `offset_token` is a safer option that will instruct Redpanda Connect to use its default at-least-once delivery model instead. ```yaml input: redpanda: topics: ["my_topic_going_to_snow"] consumer_group: "redpanda_connect_to_snowflake" # We want very large batches - each batch will be sent to Snowflake individually # so to optimize query performance we want as big of files as we have memory for fetch_max_bytes: 100MiB fetch_min_bytes: 50MiB partition_buffer_bytes: 100MiB pipeline: processors: - schema_registry_decode: url: "redpanda.example.com:8081" basic_auth: enabled: true username: MY_USER_NAME password: "${TODO}" output: fallback: - snowflake_streaming: # To ensure that we write an ordered stream each partition in kafka gets its own # channel. channel_name: "partition-${!@kafka_partition}" # Ensure that our offsets are lexicographically sorted in string form by padding with # leading zeros offset_token: offset-${!"%016X".format(@kafka_offset)} account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MYTABLE" private_key_file: "my/private/key.p8" schema_evolution: enabled: true # In order to prevent delivery orders from messing with the order of delivered records # it's important that failures are immediately sent to a dead letter queue and not retried # to Snowflake. See the ordering documentation for the "redpanda" input for more details. - retry: output: redpanda: topic: "dead_letter_queue" ``` ### [](#http-server-to-push-data-to-snowflake)HTTP Server to push data to Snowflake This example demonstrates how to create an HTTP server input that can receive HTTP PUT requests with JSON payloads, that are buffered locally then written to Snowflake in batches. > 📝 **NOTE** > > This example uses a buffer to respond to the HTTP request immediately, so it’s possible that failures to deliver data could result in data loss. See the documentation about [buffers](https://docs.redpanda.com/connect/components/buffers/memory/) for more information, or remove the buffer entirely to respond to the HTTP request only once the data is written to Snowflake. ```yaml input: http_server: path: /snowflake buffer: memory: # Max inflight data before applying backpressure limit: 524288000 # 50MiB # Batching policy, influences how large the generated files sent to Snowflake are batch_policy: enabled: true byte_size: 33554432 # 32MiB period: "10s" output: snowflake_streaming: account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MYTABLE" private_key_file: "my/private/key.p8" # By default there is only a single channel per output table allowed # if we want to have multiple Redpanda Connect streams writing data # then we need a unique channel prefix per stream. We'll use the host # name to get unique prefixes in this example. channel_prefix: "snowflake-channel-for-${HOST}" schema_evolution: enabled: true ``` See also: [Ingest data into Snowflake cookbook](https://docs.redpanda.com/connect/cookbooks/snowflake_ingestion/) --- # Page 197: socket **URL**: https://docs.redpanda.com/connect/components/outputs/socket.md --- # socket --- title: socket latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/socket page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/socket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/socket.adoc description: Connects to a (tcp/udp/unix) server and sends a continuous stream of data, dividing messages according to the specified codec. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Connects to a (tcp/udp/unix) server and sends a continuous stream of data, dividing messages according to the specified codec. #### Common ```yml output: label: "" socket: network: "" # No default (required) address: "" # No default (required) codec: lines ``` #### Advanced ```yml output: label: "" socket: network: "" # No default (required) address: "" # No default (required) codec: lines tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` ## [](#fields)Fields ### [](#address)`address` The address to connect to. **Type**: `string` ```yaml # Examples: address: /tmp/benthos.sock # --- address: 127.0.0.1:6000 ``` ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. **Type**: `string` **Default**: `lines` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | lines | Append each message to the output stream followed by a line break. | | delim:x | Append each message to the output stream followed by a custom delimiter. | ```yaml # Examples: codec: lines # --- codec: "delim:\t" # --- codec: delim:foobar ``` ### [](#network)`network` A network type to connect as. **Type**: `string` **Options**: `unix`, `tcp`, `udp` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 198: splunk_hec **URL**: https://docs.redpanda.com/connect/components/outputs/splunk_hec.md --- # splunk_hec --- title: splunk_hec latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/splunk_hec page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/splunk_hec.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/splunk_hec.adoc description: Publishes messages to a Splunk HTTP Endpoint Collector (HEC). page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Publishes messages to a Splunk HTTP Endpoint Collector (HEC). Introduced in version 4.30.0. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/connect/configuration/batching/). ### Common ```yml output: label: "" splunk_hec: url: "" # No default (required) token: "" # No default (required) gzip: false event_host: "" # No default (optional) event_source: "" # No default (optional) event_sourcetype: "" # No default (optional) event_index: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` ### Advanced ```yml output: label: "" splunk_hec: url: "" # No default (required) token: "" # No default (required) gzip: false event_host: "" # No default (optional) event_source: "" # No default (optional) event_sourcetype: "" # No default (optional) event_index: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#event_host)`event_host` Set the host value to assign to the event data. Overrides existing host field if present. **Type**: `string` ### [](#event_index)`event_index` Set the index value to assign to the event data. Overrides existing index field if present. **Type**: `string` ### [](#event_source)`event_source` Set the source value to assign to the event data. Overrides existing source field if present. **Type**: `string` ### [](#event_sourcetype)`event_sourcetype` Set the sourcetype value to assign to the event data. Overrides existing sourcetype field if present. **Type**: `string` ### [](#gzip)`gzip` Enable gzip compression **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. Requires version 4.31.0 or later. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#token)`token` A bot token used for authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#url)`url` Full HTTP Endpoint Collector (HEC) URL. **Type**: `string` ```yaml # Examples: url: https://foobar.splunkcloud.com/services/collector/event ``` --- # Page 199: sql_insert **URL**: https://docs.redpanda.com/connect/components/outputs/sql_insert.md --- # sql_insert --- title: sql_insert latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/sql_insert page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sql_insert.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sql_insert.adoc description: Inserts a row into an SQL database for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Inserts a row into an SQL database for each message. Introduced in version 3.59.0. #### Common ```yml output: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) prefix: "" # No default (optional) suffix: "" # No default (optional) options: [] # No default (optional) max_in_flight: 64 init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml output: sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ id, name, topic ] args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ## [](#dynamic-sql-operations)Dynamic SQL operations The `table` and `columns` fields are static strings that do not support Bloblang interpolation. For dynamic table names, dynamic column lists, DELETE operations, or any other SQL that `sql_insert` cannot express, use the [`sql_raw` output](https://docs.redpanda.com/connect/components/outputs/sql_raw/) instead. There is no dedicated `sql_delete` output. To delete rows, use `sql_raw` with a DELETE statement: ```yaml output: sql_raw: driver: postgres dsn: postgres://user:pass@localhost:5432/mydb?sslmode=disable query: "DELETE FROM my_table WHERE id = $1" args_mapping: root = [ this.id ] ``` To insert into a table determined at runtime, use `sql_raw` with `unsafe_dynamic_query: true`, which enables Bloblang interpolation in the `query` field. > ⚠️ **CAUTION** > > Interpolating user-supplied values into a query can introduce SQL injection risks. Always validate or sanitize the interpolated value beforehand. ```yaml output: sql_raw: driver: postgres dsn: postgres://user:pass@localhost:5432/mydb?sslmode=disable unsafe_dynamic_query: true query: 'INSERT INTO ${! this.table_name } (id, value) VALUES ($1, $2)' args_mapping: root = [ this.id, this.value ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of columns specified. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#columns)`columns[]` A list of columns to insert. **Type**: `array` ```yaml # Examples: columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to write in parallel. **Type**: `int` **Default**: `64` ### [](#options)`options[]` A list of keyword options to add before the INTO clause of the query. Requires version 4.40.0 or later. **Type**: `array` ```yaml # Examples: options: - DELAYED - IGNORE ``` ### [](#prefix)`prefix` An optional prefix to prepend to the insert query (before INSERT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the insert query. **Type**: `string` ```yaml # Examples: suffix: ON CONFLICT (name) DO NOTHING ``` ### [](#table)`table` The table to insert to. **Type**: `string` ```yaml # Examples: table: foo ``` --- # Page 200: sql_raw **URL**: https://docs.redpanda.com/connect/components/outputs/sql_raw.md --- # sql_raw --- title: sql_raw latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/sql_raw page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sql_raw.adoc description: Executes an arbitrary SQL query for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes an arbitrary SQL query for each message. Introduced in version 3.65.0. When multiple queries are configured, all messages within a batch are executed in a single database transaction. Single-query configurations execute each message individually without a transaction, preserving per-message error granularity. When consuming from Kafka, messages are automatically ordered by partition within the transaction, allowing `max_in_flight` > 1 to parallelize across partitions while preserving consume order within each partition. Messages without `kafka_partition` metadata default to partition 0. #### Common ```yml output: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) args_mapping: "" # No default (optional) queries: [] # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) unsafe_dynamic_query: false args_mapping: "" # No default (optional) queries: [] # No default (optional) max_in_flight: 64 init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` For some scenarios where you might use this output, see [Examples](#examples). ## [](#batch-execution-and-ordered-writes)Batch execution and ordered writes Configure a `batching` policy to accumulate messages and write them together. When you set more than one query (see [Conditional queries](#conditional-queries)), all messages in a batch run in a single database transaction, which reduces round-trips to the database. A single-query configuration runs each message individually without a transaction, preserving per-message error granularity. When consuming from Redpanda or Kafka, this output orders messages by partition within the transaction. This lets you raise `max_in_flight` above `1` to parallelize writes across partitions while preserving consume order within each partition, so you can scale throughput without breaking change-data-capture ordering. Messages without a `kafka_partition` metadata field are treated as partition `0`. This makes `sql_raw` suitable for high-throughput sink pipelines, including CDC, that previously required `max_in_flight: 1` to stay ordered. ## [](#conditional-queries)Conditional queries Use the `queries` field to route each message to a different SQL statement based on its content or metadata, without a preprocessing `mapping` step or `unsafe_dynamic_query`. Set a `when` condition (a Bloblang expression) on each entry: the first query whose `when` evaluates to `true`, or the first query with no `when`, runs for that message. If none of your queries have a `when` condition, every query runs for each message within the same transaction. Because an unconditioned query always matches, Redpanda Connect lints against configuring one before later queries in the list. For example, route change-data-capture tombstones to a `DELETE` and everything else to an upsert: ```yaml output: sql_raw: driver: postgres dsn: "${DSN}" max_in_flight: 8 batching: count: 100 period: 100ms queries: - when: 'root = meta("kafka_tombstone_message") == "true"' query: 'DELETE FROM orders WHERE id = $1' args_mapping: 'root = [ meta("kafka_key").parse_json().id ]' - query: | INSERT INTO orders (id, name, updated_at) VALUES ($1, $2, $3) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at args_mapping: 'root = [ this.id, this.name, this.updated_at ]' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#max_in_flight)`max_in_flight` The maximum number of batches to send in parallel at any given time. When multiple queries are configured and you consume from Redpanda or Kafka, messages are ordered by partition within each transaction, so you can keep this above `1` to parallelize writes across partitions while preserving consume order within each partition. Messages without a `kafka_partition` metadata field are treated as partition `0`. **Type**: `int` **Default**: `64` ### [](#queries)`queries[]` A list of database statements to run in addition to the main `query`. When a `when` condition is specified on entries, the first query whose condition evaluates to `true` (or that has no condition) is executed for each message. When no `when` conditions are present, all queries are executed for each message within a single transaction. Requires version 4.45.0 or later. **Type**: `array` ### [](#queries-args_mapping)`queries[].args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#queries-query)`queries[].query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ### [](#queries-when)`queries[].when` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that, when set, is evaluated for each message to determine whether to execute this query. The mapping should return a boolean value. The first query in the list whose `when` condition evaluates to `true` (or that has no `when` condition) is executed. This enables conditional query routing based on message content or metadata without requiring `unsafe_dynamic_query`. Requires version 4.87.0 or later. **Type**: `string` ```yaml # Examples: when: root = meta("kafka_tombstone_message") == "true" # --- when: root = this.operation == "delete" ``` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) in the query. Make sure your queries are defended against injection attacks. Requires version 4.8.0 or later. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml output: sql_raw: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb query: "INSERT INTO footable (id, name, topic) VALUES (?, ?, ?);" args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ### [](#dynamically-creating-tables-postgresql)Dynamically Creating Tables (PostgreSQL) Here we dynamically create output tables transactionally with inserting a record into the newly created table. ```yaml output: processors: - mapping: | root = this # Prevent SQL injection when using unsafe_dynamic_query meta table_name = "\"" + metadata("table_name").replace_all("\"", "\"\"") + "\"" sql_raw: driver: postgres dsn: postgres://localhost/postgres unsafe_dynamic_query: true queries: - query: | CREATE TABLE IF NOT EXISTS ${!metadata("table_name")} (id varchar primary key, document jsonb); - query: | INSERT INTO ${!metadata("table_name")} (id, document) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET document = EXCLUDED.document; args_mapping: | root = [ this.id, this.document.string() ] ``` ### [](#conditional-cdc-queries-postgresql)Conditional CDC Queries (PostgreSQL) Route messages to different SQL operations based on message metadata. Tombstone messages trigger a DELETE, while all other messages perform an upsert. All operations within a batch execute in a single transaction, ordered by Kafka partition. ```yaml output: sql_raw: driver: postgres dsn: postgres://localhost/postgres max_in_flight: 8 batching: count: 100 period: 100ms queries: - when: 'root = meta("kafka_tombstone_message") == "true"' query: 'DELETE FROM users WHERE id = $1' args_mapping: 'root = [this.id]' - query: | INSERT INTO users (id, name, updated_at) VALUES ($1, $2, $3) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at args_mapping: 'root = [this.id, this.name, this.updated_at]' ``` --- # Page 201: sql **URL**: https://docs.redpanda.com/connect/components/outputs/sql.md --- # sql --- title: sql latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/sql page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sql.adoc description: Executes an arbitrary SQL query for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **WARNING: Deprecated** > > Deprecated > > This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). Executes an arbitrary SQL query for each message. Introduced in version 3.65.0. #### Common ```yml output: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#alternatives)Alternatives For basic inserts use the [`sql_insert`](https://docs.redpanda.com/connect/components/outputs/sql_insert/) output. For more complex queries use the [`sql_raw`](https://docs.redpanda.com/connect/components/outputs/sql_raw/) output. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#data_source_name)`data_source_name` Data source name. **Type**: `string` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to write in parallel. **Type**: `int` **Default**: `64` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` --- # Page 202: stdout **URL**: https://docs.redpanda.com/connect/components/outputs/stdout.md --- # stdout --- title: stdout latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/stdout page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/stdout.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/stdout.adoc description: Prints messages to stdout as a continuous stream of data. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Prints messages to stdout as a continuous stream of data. #### Common ```yml output: label: "" stdout: codec: lines ``` #### Advanced ```yml output: label: "" stdout: codec: lines ``` ## [](#fields)Fields ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. Requires version 3.46.0 or later. **Type**: `string` **Default**: `lines` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | lines | Append each message to the output stream followed by a line break. | | delim:x | Append each message to the output stream followed by a custom delimiter. | ```yaml # Examples: codec: lines # --- codec: "delim:\t" # --- codec: delim:foobar ``` --- # Page 203: subprocess **URL**: https://docs.redpanda.com/connect/components/outputs/subprocess.md --- # subprocess --- title: subprocess latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/subprocess page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/subprocess.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/subprocess.adoc description: Executes a command, runs it as a subprocess, and writes messages to it over stdin. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is in beta. Its behavior might change in a future release. Executes a command, runs it as a subprocess, and writes messages to it over stdin. Messages are written according to a specified codec. The process is expected to terminate gracefully when stdin is closed. If the subprocess exits unexpectedly then Redpanda Connect will log anything printed to stderr and will log the exit code, and will attempt to execute the command again until success. The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory. #### Common ```yml output: label: "" subprocess: name: "" # No default (required) args: [] codec: lines ``` #### Advanced ```yml output: label: "" subprocess: name: "" # No default (required) args: [] codec: lines ``` ## [](#fields)Fields ### [](#args)`args[]` A list of arguments to provide the command. **Type**: `array` **Default**: `[]` ### [](#codec)`codec` The way in which messages should be written to the subprocess. **Type**: `string` **Default**: `lines` **Options**: `lines` ### [](#name)`name` The command to execute as a subprocess. **Type**: `string` --- # Page 204: switch **URL**: https://docs.redpanda.com/connect/components/outputs/switch.md --- # switch --- title: switch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/switch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/switch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/switch.adoc description: The switch output type allows you to route messages to different outputs based on their contents. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- The switch output type allows you to route messages to different outputs based on their contents. Messages that do not pass the check of a single output case are effectively dropped. In order to prevent this outcome set the field [`strict_mode`](#strict_mode) to `true`, in which case messages that do not pass at least one case are considered failed and will be nacked and/or reprocessed depending on your input. #### Common ```yml output: label: "" switch: retry_until_success: false cases: [] # No default (required) ``` #### Advanced ```yml output: label: "" switch: retry_until_success: false strict_mode: false cases: [] # No default (required) ``` ## [](#examples)Examples ### [](#basic-multiplexing)Basic Multiplexing The most common use for a switch output is to multiplex messages across a range of output destinations. The following config checks the contents of the field `type` of messages and sends `foo` type messages to an `amqp_1` output, `bar` type messages to a `gcp_pubsub` output, and everything else to a `redis_streams` output. Outputs can have their own processors associated with them, and in this example the `redis_streams` output has a processor that enforces the presence of a type field before sending it. ```yaml output: switch: cases: - check: this.type == "foo" output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - check: this.type == "bar" output: gcp_pubsub: project: dealing_with_mike topic: mikes_bars - output: redis_streams: url: tcp://localhost:6379 stream: everything_else processors: - mapping: | root = this root.type = this.type | "unknown" ``` ### [](#control-flow)Control Flow The `continue` field allows messages that have passed a case to be tested against the next one also. This can be useful when combining non-mutually-exclusive case checks. In the following example a message that passes both the check of the first case as well as the second will be routed to both. ```yaml output: switch: cases: - check: 'this.user.interests.contains("walks").catch(false)' output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/people_what_think_good continue: true - check: 'this.user.dislikes.contains("videogames").catch(false)' output: gcp_pubsub: project: people topic: that_i_dont_want_to_hang_with ``` ## [](#fields)Fields ### [](#cases)`cases[]` A list of switch cases, outlining outputs that can be routed to. **Type**: `array` ```yaml # Examples: cases: - check: this.urls.contains("http://benthos.dev") continue: true output: cache: key: ${!json("id")} target: foo - output: s3: bucket: bar path: ${!json("id")} ``` ### [](#cases-check)`cases[].check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should be routed to the case output. If left empty the case always passes. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") ``` ### [](#cases-continue)`cases[].continue` Indicates whether, if this case passes for a message, the next case should also be tested. **Type**: `bool` **Default**: `false` ### [](#cases-output)`cases[].output` An [output](https://docs.redpanda.com/connect/components/outputs/about/) for messages that pass the check to be routed to. **Type**: `output` ### [](#retry_until_success)`retry_until_success` If a selected output fails to send a message this field determines whether it is reattempted indefinitely. If set to false the error is instead propagated back to the input level. If a message can be routed to >1 outputs it is usually best to set this to true in order to avoid duplicate messages being routed to an output. **Type**: `bool` **Default**: `false` ### [](#strict_mode)`strict_mode` This field determines whether an error should be reported if no condition is met. If set to true, an error is propagated back to the input level. The default behavior is false, which will drop the message. **Type**: `bool` **Default**: `false` --- # Page 205: sync_response **URL**: https://docs.redpanda.com/connect/components/outputs/sync_response.md --- # sync_response --- title: sync_response latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/sync_response page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sync_response.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sync_response.adoc description: Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type. ## Common ```yml output: label: "" sync_response: {} ``` ## Advanced ```yml output: label: "" sync_response: {} ``` For most inputs this mechanism is ignored entirely, in which case the sync response is dropped without penalty. It is therefore safe to use this output even when combining input types that might not have support for sync responses. An example of an input able to utilize this is the `http_server`. It is safe to combine this output with others using broker types. For example, with the `http_server` input we could send the payload to a Kafka topic and also send a modified payload back with: ```yaml input: http_server: path: /post output: broker: pattern: fan_out outputs: - kafka: addresses: [ TODO:9092 ] topic: foo_topic - sync_response: {} processors: - mapping: 'root = content().uppercase()' ``` Using the above example and posting the message 'hello world' to the endpoint `/post` Redpanda Connect would send it unchanged to the topic `foo_topic` and also respond with 'HELLO WORLD'. For more information please read [synchronous responses](https://docs.redpanda.com/connect/guides/sync_responses/). --- # Page 206: timeplus **URL**: https://docs.redpanda.com/connect/components/outputs/timeplus.md --- # timeplus --- title: timeplus latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/timeplus page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/timeplus.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/timeplus.adoc description: Sends message to a Timeplus Enterprise stream via ingest endpoint page-git-created-date: "2024-11-05" page-git-modified-date: "2026-10-10" --- Sends message to a Timeplus Enterprise stream via ingest endpoint This output can send message to Timeplus Enterprise Cloud, Timeplus Enterprise (self-hosted) or directly to timeplusd. This output accepts structured message only. It also expects all message contains the same keys and matches the schema of the destination stream. If the upstream source or pipeline returns unstructured message such as string, please refer to the "Unstructured message" example. #### Common ```yml output: label: "" timeplus: target: timeplus url: https://us-west-2.timeplus.cloud workspace: "" # No default (optional) stream: "" # No default (required) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml output: label: "" timeplus: target: timeplus url: https://us-west-2.timeplus.cloud workspace: "" # No default (optional) stream: "" # No default (required) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#examples)Examples ### [](#to-timeplus-enterprise-cloud)To Timeplus Enterprise Cloud You will need to create API Key on Timeplus Enterprise Cloud Web console first and then set the `apikey` field. ```yaml output: timeplus: workspace: my_workspace_id stream: mystream apikey: ``` ### [](#to-timeplus-enterprise-self-hosted)To Timeplus Enterprise (self-hosted) For self-hosted Timeplus Enterprise, you will need to specify the username and password as well as the URL of the App server ```yaml output: timeplus: url: http://localhost:8000 workspace: my_workspace_id stream: mystream username: username password: pw ``` ### [](#to-timeplusd)To Timeplusd This output writes to Timeplusd via HTTP so make sure you specify the HTTP port of the Timeplusd. ```yaml output: timeplus: url: http://localhost:3218 stream: mystream username: username password: pw ``` ### [](#unstructured-message)Unstructured message If the upstream source or pipeline returns unstructured message such as string, you can leverage the output processors to wrap it into a structured message and then pass it to the output. This example create a structured message with `raw` field and store the original string content into this field. You can modify the name of this `raw` field to whatever you want. Please make sure the destination stream contains such field ```yaml output: timeplus: workspace: my_workspace_id stream: mystream apikey: processors: - mapping: | root = {} root.raw = content().string() ``` ## [](#fields)Fields ### [](#apikey)`apikey` The API key. Required if you are sending message to Timeplus Enterprise Cloud > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` The password. Required if you are sending message to Timeplus Enterprise (self-hosted) or timeplusd > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#stream)`stream` The name of the stream. Make sure the schema of the stream matches the input **Type**: `string` ### [](#target)`target` The destination type, either Timeplus Enterprise or timeplusd **Type**: `string` **Default**: `timeplus` **Options**: `timeplus`, `timeplusd` ### [](#url)`url` The url should always include schema and host. **Type**: `string` **Default**: `[https://us-west-2.timeplus.cloud](https://us-west-2.timeplus.cloud)` ```yaml # Examples: url: http://localhost:8000 # --- url: http://127.0.0.1:3218 ``` ### [](#username)`username` The username. Required if you are sending message to Timeplus Enterprise (self-hosted) or timeplusd **Type**: `string` ### [](#workspace)`workspace` ID of the workspace. Required if target is `timeplus`. **Type**: `string` --- # Page 207: websocket **URL**: https://docs.redpanda.com/connect/components/outputs/websocket.md --- # websocket --- title: websocket latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/websocket page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/websocket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/websocket.adoc description: Sends messages to an HTTP server via a websocket connection. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends messages to an HTTP server via a websocket connection. #### Common ```yml output: label: "" websocket: url: "" # No default (required) ``` #### Advanced ```yml output: label: "" websocket: url: "" # No default (required) proxy_url: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#fields)Fields ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. Requires version 4.30.0 or later. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. **Type**: `string` --- # Page 208: zmq4 **URL**: https://docs.redpanda.com/connect/components/outputs/zmq4.md --- # zmq4 --- title: zmq4 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: outputs/zmq4 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/zmq4.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/zmq4.adoc description: Writes messages to a ZeroMQ socket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Writes messages to a ZeroMQ socket. #### Common ```yml output: label: "" zmq4: urls: [] # No default (required) bind: true socket_type: "" # No default (required) ``` #### Advanced ```yml output: label: "" zmq4: urls: [] # No default (required) bind: true socket_type: "" # No default (required) high_water_mark: 0 poll_timeout: 5s ``` > 📝 **NOTE** > > This component is available only in builds of Redpanda Connect that enable cgo. The standard release binaries, the Docker images, and the standard `rpk connect` plugin don’t include it. To use it, do one of the following: > > - Download the `redpanda-connect-cgo__linux_amd64` archive from the [GitHub releases page](https://github.com/redpanda-data/connect/releases). This archive is available for Linux AMD64 only. > > - Build Redpanda Connect from source with `CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata ./cmd/redpanda-connect`. The build needs a C compiler. > > > The cgo archive and the source build both need the ZeroMQ library (`libzmq`), because they include the `zmq4` components. ## [](#fields)Fields ### [](#bind)`bind` Whether to bind to the URLs and wait for peers to connect, instead of connecting to them. **Type**: `bool` **Default**: `true` ### [](#high_water_mark)`high_water_mark` The message high water mark to use. **Type**: `int` **Default**: `0` ### [](#poll_timeout)`poll_timeout` The poll timeout to use. **Type**: `string` **Default**: `5s` ### [](#socket_type)`socket_type` The socket type to connect as. **Type**: `string` **Options**: `PUSH`, `PUB` ### [](#urls)`urls[]` A list of ZeroMQ endpoints to connect to, or to bind when `bind` is `true`, using a transport such as `tcp://`, `ipc://`, or `inproc://`. If an item in the list contains commas, it is split into multiple endpoints. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:5556" ``` --- # Page 209: Processors **URL**: https://docs.redpanda.com/connect/components/processors/about.md --- # Processors --- title: Processors latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/about.adoc description: Learn how processors work in Redpanda Connect. Processors are functions that mutate, filter, or enrich messages passing through a pipeline. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Redpanda Connect processors are functions applied to messages passing through a pipeline. The function signature allows a processor to mutate or drop messages depending on the content of the message. There are many types on offer but the most powerful are the [`mapping`](https://docs.redpanda.com/connect/components/processors/mapping/) and [`mutation`](https://docs.redpanda.com/connect/components/processors/mutation/) processors. Processors are set via config, and depending on where in the config they are placed they will be run either immediately after a specific input (set in the input section), on all messages (set in the pipeline section) or before a specific output (set in the output section). Most processors apply to all messages and can be placed in the pipeline section: ```yaml pipeline: threads: 1 processors: - label: my_cool_mapping mapping: | root.message = this root.meta.link_count = this.links.length() ``` The `threads` field in the pipeline section determines how many parallel processing threads are created. You can read more about parallel processing in the [pipeline guide](https://docs.redpanda.com/connect/configuration/processing_pipelines/). ## [](#labels)Labels Processors have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple nested processors, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](https://docs.redpanda.com/connect/components/metrics/about/). ## [](#error-handling)Error handling Some processors have conditions whereby they might fail. Rather than throw these messages into the abyss Redpanda Connect still attempts to send these messages onwards, and has mechanisms for filtering, recovering or dead-letter queuing messages that have failed which can be read about [here](https://docs.redpanda.com/connect/configuration/error_handling/). ### [](#error-logs)Error logs Errors that occur during processing can be roughly separated into two groups; those that are unexpected intermittent errors such as connectivity problems, and those that are logical errors such as bad input data or unmatched schemas. All processing errors result in the messages being flagged as failed, [error metrics](https://docs.redpanda.com/connect/components/metrics/about/) increasing for the given errored processor, and debug level logs being emitted that describe the error. Only errors that are known to be intermittent are also logged at the error level. The reason for this behavior is to prevent noisy logging in cases where logical errors are expected and will likely be [handled in config](https://docs.redpanda.com/connect/configuration/error_handling/). However, this can also sometimes make it easy to miss logical errors in your configs when they lack error handling. If you suspect you are experiencing processing errors and do not wish to add error handling yet then a quick and easy way to expose those errors is to enable debug level logs with the cli flag `--log.level=debug` or by setting the level in config: ```yaml logger: level: DEBUG ``` ## [](#using-processors-as-outputs)Using processors as outputs It might be the case that a processor that results in a side effect, such as the [`sql_insert`](https://docs.redpanda.com/connect/components/processors/sql_insert/) or [`redis`](https://docs.redpanda.com/connect/components/processors/redis/) processors, is the only side effect of a pipeline, and therefore could be considered the output. In such cases it’s possible to place these processors within a [`reject` output](https://docs.redpanda.com/connect/components/outputs/reject/) so that they behave the same as regular outputs, where success results in dropping the message with an acknowledgement and failure results in a nack (or retry): ```yaml output: reject: 'failed to send data: ${! error() }' processors: - try: - redis: url: tcp://localhost:6379 command: sadd args_mapping: 'root = [ this.key, this.value ]' - mapping: root = deleted() ``` The way this works is that if your processor with the side effect (`redis` in this case) succeeds then the final `mapping` processor deletes the message which results in an acknowledgement. If the processor fails then the `try` block exits early without executing the `mapping` processor and instead the message is routed to the `reject` output, which nacks the message with an error message containing the error obtained from the `redis` processor. ## [](#categories)Categories ### Parsing Processors that specialize in translating messages from one format to another. [**ZIP**](https://docs.redpanda.com/connect/components/processors/archive/) [**avro**](https://docs.redpanda.com/connect/components/processors/avro/) [**bloblang**](https://docs.redpanda.com/connect/components/processors/bloblang/) [**compress**](https://docs.redpanda.com/connect/components/processors/compress/) [**decompress**](https://docs.redpanda.com/connect/components/processors/decompress/) [**grok**](https://docs.redpanda.com/connect/components/processors/grok/) [**mapping**](https://docs.redpanda.com/connect/components/processors/mapping/) [**msgpack**](https://docs.redpanda.com/connect/components/processors/msgpack/) [**mutation**](https://docs.redpanda.com/connect/components/processors/mutation/) [**parquet\_decode**](https://docs.redpanda.com/connect/components/processors/parquet_decode/) [**parquet\_encode**](https://docs.redpanda.com/connect/components/processors/parquet_encode/) [**parse\_log**](https://docs.redpanda.com/connect/components/processors/parse_log/) [**protobuf**](https://docs.redpanda.com/connect/components/processors/protobuf/) [**schema\_registry\_decode**](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) [**schema\_registry\_encode**](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/) [**string\_split**](https://docs.redpanda.com/connect/components/processors/string_split/) [**ZIP**](https://docs.redpanda.com/connect/components/processors/unarchive/) [**xml**](https://docs.redpanda.com/connect/components/processors/xml/) ### Utility Processors that provide general utility or do not fit in another category. [**ZIP**](https://docs.redpanda.com/connect/components/processors/archive/) [**benchmark**](https://docs.redpanda.com/connect/components/processors/benchmark/) [**bounds\_check**](https://docs.redpanda.com/connect/components/processors/bounds_check/) [**cached**](https://docs.redpanda.com/connect/components/processors/cached/) [**crash**](https://docs.redpanda.com/connect/components/processors/crash/) [**dedupe**](https://docs.redpanda.com/connect/components/processors/dedupe/) [**log**](https://docs.redpanda.com/connect/components/processors/log/) [**metric**](https://docs.redpanda.com/connect/components/processors/metric/) [**rate\_limit**](https://docs.redpanda.com/connect/components/processors/rate_limit/) [**redpanda\_data\_transform**](https://docs.redpanda.com/connect/components/processors/redpanda_data_transform/) [**resource**](https://docs.redpanda.com/connect/components/processors/resource/) [**select\_parts**](https://docs.redpanda.com/connect/components/processors/select_parts/) [**sleep**](https://docs.redpanda.com/connect/components/processors/sleep/) [**split**](https://docs.redpanda.com/connect/components/processors/split/) [**sync\_response**](https://docs.redpanda.com/connect/components/processors/sync_response/) [**ZIP**](https://docs.redpanda.com/connect/components/processors/unarchive/) [**wasm**](https://docs.redpanda.com/connect/components/processors/wasm/) ### Mapping Processors that specialize in restructuring messages. [**awk**](https://docs.redpanda.com/connect/components/processors/awk/) [**bloblang**](https://docs.redpanda.com/connect/components/processors/bloblang/) [**javascript**](https://docs.redpanda.com/connect/components/processors/javascript/) [**jmespath**](https://docs.redpanda.com/connect/components/processors/jmespath/) [**jq**](https://docs.redpanda.com/connect/components/processors/jq/) [**JSON Schema**](https://docs.redpanda.com/connect/components/processors/json_schema/) [**mapping**](https://docs.redpanda.com/connect/components/processors/mapping/) [**mutation**](https://docs.redpanda.com/connect/components/processors/mutation/) ### AI [**Amazon**](https://docs.redpanda.com/connect/components/processors/aws_bedrock_chat/) [**Amazon**](https://docs.redpanda.com/connect/components/processors/aws_bedrock_embeddings/) [**cohere\_chat**](https://docs.redpanda.com/connect/components/processors/cohere_chat/) [**cohere\_embeddings**](https://docs.redpanda.com/connect/components/processors/cohere_embeddings/) [**cohere\_rerank**](https://docs.redpanda.com/connect/components/processors/cohere_rerank/) [**GCP Vertex AI**](https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_chat/) [**Google Cloud**](https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_embeddings/) [**ollama\_chat**](https://docs.redpanda.com/connect/components/processors/ollama_chat/) [**ollama\_embeddings**](https://docs.redpanda.com/connect/components/processors/ollama_embeddings/) [**ollama\_moderation**](https://docs.redpanda.com/connect/components/processors/ollama_moderation/) [**openai\_chat\_completion**](https://docs.redpanda.com/connect/components/processors/openai_chat_completion/) [**openai\_embeddings**](https://docs.redpanda.com/connect/components/processors/openai_embeddings/) [**openai\_image\_generation**](https://docs.redpanda.com/connect/components/processors/openai_image_generation/) [**openai\_speech**](https://docs.redpanda.com/connect/components/processors/openai_speech/) [**openai\_transcription**](https://docs.redpanda.com/connect/components/processors/openai_transcription/) [**openai\_translation**](https://docs.redpanda.com/connect/components/processors/openai_translation/) [**qdrant**](https://docs.redpanda.com/connect/components/processors/qdrant/) [**text\_chunker**](https://docs.redpanda.com/connect/components/processors/text_chunker/) ### Integration Processors that interact with external services. [**Amazon**](https://docs.redpanda.com/connect/components/processors/aws_dynamodb_partiql/) [**AWS Lambda**](https://docs.redpanda.com/connect/components/processors/aws_lambda/) [**cache**](https://docs.redpanda.com/connect/components/processors/cache/) [**command**](https://docs.redpanda.com/connect/components/processors/command/) [**couchbase**](https://docs.redpanda.com/connect/components/processors/couchbase/) [**GCP BigQuery**](https://docs.redpanda.com/connect/components/processors/gcp_bigquery_select/) [**http**](https://docs.redpanda.com/connect/components/processors/http/) [**redis**](https://docs.redpanda.com/connect/components/processors/redis/) [**Redis Script**](https://docs.redpanda.com/connect/components/processors/redis_script/) [**schema\_registry\_decode**](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) [**schema\_registry\_encode**](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/) [**SQL**](https://docs.redpanda.com/connect/components/processors/sql_insert/) [**SQL**](https://docs.redpanda.com/connect/components/processors/sql_raw/) [**SQL**](https://docs.redpanda.com/connect/components/processors/sql_select/) [**subprocess**](https://docs.redpanda.com/connect/components/processors/subprocess/) ### Azure [**Microsoft Azure**](https://docs.redpanda.com/connect/components/processors/azure_cosmosdb/) ### Composition Higher level processors that compose other processors and modify their behavior. [**branch**](https://docs.redpanda.com/connect/components/processors/branch/) [**catch**](https://docs.redpanda.com/connect/components/processors/catch/) [**for\_each**](https://docs.redpanda.com/connect/components/processors/for_each/) [**group\_by**](https://docs.redpanda.com/connect/components/processors/group_by/) [**group\_by\_value**](https://docs.redpanda.com/connect/components/processors/group_by_value/) [**insert\_part**](https://docs.redpanda.com/connect/components/processors/insert_part/) [**parallel**](https://docs.redpanda.com/connect/components/processors/parallel/) [**processors**](https://docs.redpanda.com/connect/components/processors/processors/) [**retry**](https://docs.redpanda.com/connect/components/processors/retry/) [**switch**](https://docs.redpanda.com/connect/components/processors/switch/) [**try**](https://docs.redpanda.com/connect/components/processors/try/) [**try\_catch**](https://docs.redpanda.com/connect/components/processors/try_catch/) [**while**](https://docs.redpanda.com/connect/components/processors/while/) [**workflow**](https://docs.redpanda.com/connect/components/processors/workflow/) ### [**Foreign Function Interface**](https://docs.redpanda.com/connect/components/processors/ffi/) ### Unstructured [**google\_drive\_download**](https://docs.redpanda.com/connect/components/processors/google_drive_download/) [**google\_drive\_list\_labels**](https://docs.redpanda.com/connect/components/processors/google_drive_list_labels/) [**google\_drive\_search**](https://docs.redpanda.com/connect/components/processors/google_drive_search/) ### Services [**MongoDB**](https://docs.redpanda.com/connect/components/processors/mongodb/) [**NATS KV**](https://docs.redpanda.com/connect/components/processors/nats_kv/) [**NATS Request Reply**](https://docs.redpanda.com/connect/components/processors/nats_request_reply/) ## [](#batching-and-multiple-part-messages)Batching and multiple-part messages All Redpanda Connect processors support multiple-part messages, which are synonymous with batches. This enables [windowed processing](https://docs.redpanda.com/connect/configuration/windowed_processing/) capabilities. Many processors are able to perform their behaviors on specific parts of a message batch, or on all parts, and have a field `parts` for specifying an array of part indexes they should apply to. If the list of target parts is empty these processors will be applied to all message parts. Part indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if part = -1 then the selected part will be the last part of the message, if part = -2 then the part before the last element will be selected, and so on. Some processors such as [`dedupe`](https://docs.redpanda.com/connect/components/processors/dedupe/) act across an entire batch, when instead we might like to perform them on individual messages of a batch. In this case the [`for_each`](https://docs.redpanda.com/connect/components/processors/for_each/) processor can be used. You can read more about batching [in this document](https://docs.redpanda.com/connect/configuration/batching/). --- # Page 210: archive **URL**: https://docs.redpanda.com/connect/components/processors/archive.md --- # archive --- title: archive latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/archive page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/archive.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/archive.adoc description: Archives all the messages of a batch into a single message according to the selected archive format. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Archives all the messages of a batch into a single message according to the selected archive format. Some archive formats (such as tar, zip) treat each archive item (message part) as a file with a path. Since message parts only contain raw data a unique path must be generated for each part. This can be done by using function interpolations on the 'path' field as described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). For types that aren’t file based (such as binary) the file field is ignored. The resulting archived message adopts the metadata of the _first_ message part of the batch. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](https://docs.redpanda.com/connect/configuration/batching/). To reverse this process use the [`unarchive` processor](https://docs.redpanda.com/connect/components/processors/unarchive/) followed by a [`split` processor](https://docs.redpanda.com/connect/components/processors/split/) to process each message individually. #### Common ```yml processor: label: "" archive: format: "" # No default (required) path: "" ``` #### Advanced ```yml processor: label: "" archive: format: "" # No default (required) path: "" ``` ## [](#fields)Fields ### [](#format)`format` The archiving format to apply. **Type**: `string` | Option | Summary | | --- | --- | | binary | Archive messages to a binary blob format. | | concatenate | Join the raw contents of each message into a single binary message. | | json_array | Attempt to parse each message as a JSON document and append the result to an array, which becomes the contents of the resulting message. | | lines | Join the raw contents of each message and insert a line break between each one. | | tar | Archive messages to a unix standard tape archive. | | zip | Archive messages to a zip file. | ### [](#path)`path` The path to set for each message in the archive (when applicable). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}-${!json("id")}.json ``` ## [](#examples)Examples ### [](#tar-archive)Tar Archive If we had JSON messages in a batch each of the form: ```json {"doc":{"id":"foo","body":"hello world 1"}} ``` And we wished to tar archive them, setting their filenames to their respective unique IDs (with the extension `.json`), our config might look like this: ```yaml pipeline: processors: - archive: format: tar path: ${!json("doc.id")}.json ``` --- # Page 211: avro **URL**: https://docs.redpanda.com/connect/components/processors/avro.md --- # avro --- title: avro latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/avro page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/avro.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/avro.adoc description: Performs Avro based operations on messages based on a schema. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs Avro based operations on messages based on a schema. > ⚠️ **WARNING** > > If you are consuming or generating messages using a schema registry service then it is likely this processor will fail as those services require messages to be prefixed with the identifier of the schema version being used. Instead, try the [`schema_registry_encode`](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/) and [`schema_registry_decode`](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) processors. ## [](#operators)Operators ### [](#to_json)`to_json` Converts Avro documents into a JSON structure. This makes it easier to manipulate the contents of the document within Benthos. The encoding field specifies how the source documents are encoded. ### [](#from_json)`from_json` Attempts to convert JSON documents into Avro documents according to the specified encoding. #### Common ```yml processor: label: "" avro: operator: "" # No default (required) encoding: textual schema: "" schema_path: "" ``` #### Advanced ```yml processor: label: "" avro: operator: "" # No default (required) encoding: textual schema: "" schema_path: "" ``` ## [](#fields)Fields ### [](#encoding)`encoding` An Avro encoding format to use for conversions to and from a schema. **Type**: `string` **Default**: `textual` **Options**: `textual`, `binary`, `single` ### [](#operator)`operator` The [operator](#operators) to execute **Type**: `string` **Options**: `to_json`, `from_json` ### [](#schema)`schema` A full Avro schema to use. **Type**: `string` **Default**: `""` ### [](#schema_path)`schema_path` The path of a schema document to apply. Use either this or the `schema` field. URLs must begin with `file://` or `http://`. Note that `file://` URLs must use absolute paths (for example, `[file:///absolute/path/to/spec.avsc](file:///absolute/path/to/spec.avsc)`); relative paths are not supported. **Type**: `string` **Default**: `""` ```yaml # Examples: schema_path: file:///path/to/spec.avsc # --- schema_path: http://localhost:8081/path/to/spec/versions/1 ``` --- # Page 212: awk **URL**: https://docs.redpanda.com/connect/components/processors/awk.md --- # awk --- title: awk latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/awk page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/awk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/awk.adoc description: Executes an AWK program on messages. This processor is very powerful as it offers a range of custom functions for querying and mutating message contents and metadata. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes an AWK program on messages. This processor is very powerful as it offers a range of [custom functions](#awk-functions) for querying and mutating message contents and metadata. Works by feeding message contents as the program input based on a chosen [codec](#codecs) and replaces the contents of each message with the result. If the result is empty (nothing is printed by the program) then the original message contents remain unchanged. Comes with a wide range of [custom functions](#awk-functions) for accessing message metadata, json fields, printing logs, etc. These functions can be overridden by functions within the program. Check out the [examples section](#examples) in order to see how this processor can be used. This processor uses [GoAWK](https://github.com/benhoyt/goawk), in order to understand the differences in how the program works you can read more about it in [goawk.differences](https://github.com/benhoyt/goawk#differences-from-awk). #### Common ```yml processor: label: "" awk: codec: "" # No default (required) program: "" # No default (required) ``` #### Advanced ```yml processor: label: "" awk: codec: "" # No default (required) program: "" # No default (required) ``` ## [](#fields)Fields ### [](#codec)`codec` A [codec](#codecs) defines how messages should be inserted into the AWK program as variables. The codec does not change which [custom Redpanda Connect functions](#awk-functions) are available. The `text` codec is the closest to a typical AWK use case. **Type**: `string` **Options**: `none`, `text`, `json` ### [](#program)`program` An AWK program to execute **Type**: `string` ## [](#examples)Examples ### [](#json-mapping-and-arithmetic)JSON Mapping and Arithmetic Because AWK is a full programming language it’s much easier to map documents and perform arithmetic with it than with other Redpanda Connect processors. For example, if we were expecting documents of the form: ```json {"doc":{"val1":5,"val2":10},"id":"1","type":"add"} {"doc":{"val1":5,"val2":10},"id":"2","type":"multiply"} ``` And we wished to perform the arithmetic specified in the `type` field, on the values `val1` and `val2` and, finally, map the result into the document, giving us the following resulting documents: ```json {"doc":{"result":15,"val1":5,"val2":10},"id":"1","type":"add"} {"doc":{"result":50,"val1":5,"val2":10},"id":"2","type":"multiply"} ``` We can do that with the following: ```yaml pipeline: processors: - awk: codec: none program: | function map_add_vals() { json_set_int("doc.result", json_get("doc.val1") + json_get("doc.val2")); } function map_multiply_vals() { json_set_int("doc.result", json_get("doc.val1") * json_get("doc.val2")); } function map_unknown(type) { json_set("error","unknown document type"); print_log("Document type not recognised: " type, "ERROR"); } { type = json_get("type"); if (type == "add") map_add_vals(); else if (type == "multiply") map_multiply_vals(); else map_unknown(type); } ``` ### [](#stuff-with-arrays)Stuff With Arrays It’s possible to iterate JSON arrays by appending an index value to the path, this can be used to do things like removing duplicates from arrays. For example, given the following input document: ```json {"path":{"to":{"foos":["one","two","three","two","four"]}}} ``` We could create a new array `foos_unique` from `foos` giving us the result: ```json {"path":{"to":{"foos":["one","two","three","two","four"],"foos_unique":["one","two","three","four"]}}} ``` With the following config: ```yaml pipeline: processors: - awk: codec: none program: | { array_path = "path.to.foos" array_len = json_length(array_path) for (i = 0; i < array_len; i++) { ele = json_get(array_path "." i) if ( ! ( ele in seen ) ) { json_append(array_path "_unique", ele) seen[ele] = 1 } } } ``` ## [](#codecs)Codecs The chosen codec determines how the contents of the message are fed into the program. Codecs only impact the input string and variables initialized for your program, they do not change the range of custom functions available. ### [](#none)`none` An empty string is fed into the program. Functions can still be used in order to extract and mutate metadata and message contents. This is useful for when your program only uses functions and doesn’t need the full text of the message to be parsed by the program, as it is significantly faster. ### [](#text)`text` The full contents of the message are fed into the program as a string, allowing you to reference tokenized segments of the message with variables ($0, $1, etc). Custom functions can still be used with this codec. This is the default codec as it behaves most similar to typical usage of the awk command line tool. ### [](#json)`json` An empty string is fed into the program, and variables are automatically initialized before execution of your program by walking the flattened JSON structure. Each value is converted into a variable by taking its full path, e.g. the object: ```json { "foo": { "bar": { "value": 10 }, "created_at": "2018-12-18T11:57:32" } } ``` Would result in the following variable declarations: ```none foo_bar_value = 10 foo_created_at = "2018-12-18T11:57:32" ``` Custom functions can also still be used with this codec. ## [](#awk-functions)AWK functions ### [](#json_get)`json_get` Signature: `json_get(path)` Attempts to find a JSON value in the input message payload by a [dot separated path](https://docs.redpanda.com/connect/configuration/field_paths/) and returns it as a string. ### [](#json_set)`json_set` Signature: `json_set(path, value)` Attempts to set a JSON value in the input message payload identified by a [dot separated path](https://docs.redpanda.com/connect/configuration/field_paths/), the value argument will be interpreted as a string. In order to set non-string values use one of the following typed varieties: - `json_set_int(path, value)` - `json_set_float(path, value)` - `json_set_bool(path, value)` ### [](#json_append)`json_append` Signature: `json_append(path, value)` Attempts to append a value to an array identified by a [dot separated path](https://docs.redpanda.com/connect/configuration/field_paths/). If the target does not exist it will be created. If the target exists but is not already an array then it will be converted into one, with its original contents set to the first element of the array. The value argument will be interpreted as a string. In order to append non-string values use one of the following typed varieties: - `json_append_int(path, value)` - `json_append_float(path, value)` - `json_append_bool(path, value)` ### [](#json_delete)`json_delete` Signature: `json_delete(path)` Attempts to delete a JSON field from the input message payload identified by a [dot separated path](https://docs.redpanda.com/connect/configuration/field_paths/). ### [](#json_length)`json_length` Signature: `json_length(path)` Returns the size of the string or array value of JSON field from the input message payload identified by a [dot separated path](https://docs.redpanda.com/connect/configuration/field_paths/). If the target field does not exist, or is not a string or array type, then zero is returned. In order to explicitly check the type of a field use `json_type`. ### [](#json_type)`json_type` Signature: `json_type(path)` Returns the type of a JSON field from the input message payload identified by a [dot separated path](https://docs.redpanda.com/connect/configuration/field_paths/). Possible values are: "string", "int", "float", "bool", "undefined", "null", "array", "object". ### [](#create_json_object)`create_json_object` Signature: `create_json_object(key1, val1, key2, val2, …​)` Generates a valid JSON object of key value pair arguments. The arguments are variadic, meaning any number of pairs can be listed. The value will always resolve to a string regardless of the value type. E.g. the following call: `create_json_object("a", "1", "b", 2, "c", "3")` Would result in this string: `\{"a":"1","b":"2","c":"3"}` ### [](#create_json_array)`create_json_array` Signature: `create_json_array(val1, val2, …​)` Generates a valid JSON array of value arguments. The arguments are variadic, meaning any number of values can be listed. The value will always resolve to a string regardless of the value type. E.g. the following call: `create_json_array("1", 2, "3")` Would result in this string: `["1","2","3"]` ### [](#metadata_set)`metadata_set` Signature: `metadata_set(key, value)` Set a metadata key for the message to a value. The value will always resolve to a string regardless of the value type. ### [](#metadata_get)`metadata_get` Signature: `metadata_get(key) string` Get the value of a metadata key from the message. ### [](#timestamp_unix)`timestamp_unix` Signature: `timestamp_unix() int` Returns the current unix timestamp (the number of seconds since 01-01-1970). ### [](#timestamp_unix-2)`timestamp_unix` Signature: `timestamp_unix(date) int` Attempts to parse a date string by detecting its format and returns the equivalent unix timestamp (the number of seconds since 01-01-1970). ### [](#timestamp_unix-3)`timestamp_unix` Signature: `timestamp_unix(date, format) int` Attempts to parse a date string according to a format and returns the equivalent unix timestamp (the number of seconds since 01-01-1970). The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. ### [](#timestamp_unix_nano)`timestamp_unix_nano` Signature: `timestamp_unix_nano() int` Returns the current unix timestamp in nanoseconds (the number of nanoseconds since 01-01-1970). ### [](#timestamp_unix_nano-2)`timestamp_unix_nano` Signature: `timestamp_unix_nano(date) int` Attempts to parse a date string by detecting its format and returns the equivalent unix timestamp in nanoseconds (the number of nanoseconds since 01-01-1970). ### [](#timestamp_unix_nano-3)`timestamp_unix_nano` Signature: `timestamp_unix_nano(date, format) int` Attempts to parse a date string according to a format and returns the equivalent unix timestamp in nanoseconds (the number of nanoseconds since 01-01-1970). The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. ### [](#timestamp_format)`timestamp_format` Signature: `timestamp_format(unix, format) string` Formats a unix timestamp. The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. The format is optional, and if omitted RFC3339 (`2006-01-02T15:04:05Z07:00`) will be used. ### [](#timestamp_format_nano)`timestamp_format_nano` Signature: `timestamp_format_nano(unixNano, format) string` Formats a unix timestamp in nanoseconds. The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. The format is optional, and if omitted RFC3339 (`2006-01-02T15:04:05Z07:00`) will be used. ### [](#print_log)`print_log` Signature: `print_log(message, level)` Prints a Redpanda Connect log message at a particular log level. The log level is optional, and if omitted the level `INFO` will be used. ### [](#base64_encode)`base64_encode` Signature: `base64_encode(data)` Encodes the input data to a base64 string. ### [](#base64_decode)`base64_decode` Signature: `base64_decode(data)` Attempts to base64-decode the input data and returns the decoded string if successful. It will emit an error otherwise. --- # Page 213: aws_bedrock_chat **URL**: https://docs.redpanda.com/connect/components/processors/aws_bedrock_chat.md --- # aws_bedrock_chat --- title: aws_bedrock_chat latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/aws_bedrock_chat page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_bedrock_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_bedrock_chat.adoc description: Generates responses to messages in a chat conversation, using the AWS Bedrock API. page-git-created-date: "2024-09-04" page-git-modified-date: "2026-10-10" --- Generates responses to messages in a chat conversation, using the AWS Bedrock API. Introduced in version 4.34.0. This processor sends prompts to your chosen large language model (LLM) and generates text from the responses, using the AWS Bedrock API. For more information, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide). #### Common ```yml processor: label: "" aws_bedrock_chat: model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) ``` #### Advanced ```yml processor: label: "" aws_bedrock_chat: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) stop: [] # No default (optional) top_p: 0 # No default (optional) ``` ## [](#fields)Fields ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to allow in the generated response. **Type**: `int` ### [](#model)`model` The model ID to use. For a full list, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). **Type**: `string` ```yaml # Examples: model: amazon.titan-text-express-v1 # --- model: anthropic.claude-3-5-sonnet-20240620-v1:0 # --- model: cohere.command-text-v14 # --- model: meta.llama3-1-70b-instruct-v1:0 # --- model: mistral.mistral-large-2402-v1:0 ``` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#stop)`stop[]` A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#temperature)`temperature` The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options. A higher value makes the model more likely to choose lower-probability options. **Type**: `float` ### [](#top_p)`top_p` The percentage of most-likely candidates that the model considers for the next token. For example, if you choose a value of `0.8`, the model selects from the top 80% of the probability distribution of tokens that could be next in the sequence. **Type**: `float` --- # Page 214: aws_bedrock_embeddings **URL**: https://docs.redpanda.com/connect/components/processors/aws_bedrock_embeddings.md --- # aws_bedrock_embeddings --- title: aws_bedrock_embeddings latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/aws_bedrock_embeddings page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_bedrock_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_bedrock_embeddings.adoc description: Computes vector embeddings on text, using the AWS Bedrock API. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Generates vector embeddings from text prompts, using the [AWS Bedrock API](https://aws.amazon.com/bedrock/). Introduced in version 4.37.0. #### Common ```yml processor: label: "" aws_bedrock_embeddings: model: "" # No default (required) text: "" # No default (optional) input_type: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" aws_bedrock_embeddings: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) model: "" # No default (required) text: "" # No default (optional) input_type: "" # No default (optional) ``` This processor sends text prompts to your chosen large language model (LLM), which generates vector embeddings for them using the AWS Bedrock API. For more information, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide). ## [](#fields)Fields ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#input_type)`input_type` Specifies the type of input passed to the model. Required by Cohere embedding models; ignored by Amazon Titan models. Requires version 4.97.0 or later. **Type**: `string` | Option | Summary | | --- | --- | | classification | Used for embeddings passed through a text classifier. | | clustering | Used for the embeddings run through a clustering algorithm. | | search_document | Used for embeddings stored in a vector database for search use-cases. | | search_query | Used for embeddings of search queries run against a vector DB to find relevant documents. | ### [](#model)`model` The ID of the model that you want to use to generate vector embeddings. For a full list, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). **Type**: `string` ```yaml # Examples: model: amazon.titan-embed-text-v1 # --- model: amazon.titan-embed-text-v2:0 # --- model: cohere.embed-english-v3 # --- model: cohere.embed-multilingual-v3 # --- model: cohere.embed-v4:0 ``` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#text)`text` The prompt you want to generate a vector embedding for. By default, the processor submits the entire payload as a string. **Type**: `string` ## [](#examples)Examples ### [](#store-embedding-vectors-in-clickhouse)Store embedding vectors in Clickhouse Compute embeddings for some generated data and store it within [Clickhouse](https://clickhouse.com/) ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - branch: request_map: | root = this.text processors: - aws_bedrock_embeddings: model: amazon.titan-embed-text-v1 result_map: | root.embeddings = this output: sql_insert: driver: clickhouse dsn: "clickhouse://localhost:9000" table: searchable_text columns: ["id", "text", "vector"] args_mapping: "root = [uuid_v4(), this.text, this.embeddings]" ``` --- # Page 215: aws_dynamodb_partiql **URL**: https://docs.redpanda.com/connect/components/processors/aws_dynamodb_partiql.md --- # aws_dynamodb_partiql --- title: aws_dynamodb_partiql latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/aws_dynamodb_partiql page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_dynamodb_partiql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_dynamodb_partiql.adoc description: Executes a PartiQL expression against a DynamoDB table for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a PartiQL expression against a DynamoDB table for each message. Introduced in version 3.48.0. Both writes or reads are supported, when the query is a read the contents of the message will be replaced with the result. This processor is more efficient when messages are pre-batched as the whole batch will be executed in a single call. #### Common ```yml processor: label: "" aws_dynamodb_partiql: query: "" # No default (required) args_mapping: "" ``` #### Advanced ```yml processor: label: "" aws_dynamodb_partiql: query: "" # No default (required) unsafe_dynamic_query: false use_batch: true args_mapping: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#examples)Examples ### [](#insert)Insert The following example inserts rows into the table footable with the columns foo, bar and baz populated with values extracted from messages: ```yaml pipeline: processors: - aws_dynamodb_partiql: query: "INSERT INTO footable VALUE {'foo':'?','bar':'?','baz':'?'}" args_mapping: | root = [ { "S": this.foo }, { "S": meta("kafka_topic") }, { "S": this.document.content }, ] ``` ### [](#query-a-gsi-for-a-single-record)Query a GSI for a single record The following example looks up a single record from the table footable using the global secondary index index\_name, matching on the field bar. BatchExecuteStatement can’t query a GSI, so use\_batch is disabled: ```yaml pipeline: processors: - aws_dynamodb_partiql: query: "SELECT * FROM \"footable\".\"index_name\" WHERE bar = ?" use_batch: false args_mapping: | root = [ { "S": this.bar }, ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that, for each message, creates a list of arguments to use with the query. **Type**: `string` **Default**: `""` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#query)`query` A PartiQL query to execute for each message. **Type**: `string` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable dynamic queries that support interpolation functions. **Type**: `bool` **Default**: `false` ### [](#use_batch)`use_batch` Whether to execute all messages in a batch as a single `BatchExecuteStatement` call. Set this to `false` to execute one `ExecuteStatement` call per message instead, which is required for PartiQL `SELECT` queries against a global secondary index (GSI), because `BatchExecuteStatement` does not support querying a GSI. Only the first result row is used when a query returns multiple items. Requires version 4.102.0 or later. **Type**: `bool` **Default**: `true` --- # Page 216: aws_lambda **URL**: https://docs.redpanda.com/connect/components/processors/aws_lambda.md --- # aws_lambda --- title: aws_lambda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/aws_lambda page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_lambda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_lambda.adoc description: Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message. Introduced in version 3.36.0. The `rate_limit` field can be used to specify a rate limit [resource](https://docs.redpanda.com/connect/components/rate_limits/about/) to cap the rate of requests across parallel components service wide. In order to map or encode the payload to a specific request body, and map the response back into the original payload instead of replacing it entirely, you can use the [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/). ## [](#error-handling)Error handling When Redpanda Connect is unable to connect to the AWS endpoint or is otherwise unable to invoke the target lambda function it will retry the request according to the configured number of retries. Once these attempts have been exhausted the failed message will continue through the pipeline with it’s contents unchanged, but flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). However, if the invocation of the function is successful but the function itself throws an error, then the message will have it’s contents updated with a JSON payload describing the reason for the failure, and a metadata field `lambda_function_error` will be added to the message allowing you to detect and handle function errors with a [`branch`](https://docs.redpanda.com/connect/components/processors/branch/): ```yaml pipeline: processors: - branch: processors: - aws_lambda: function: foo result_map: | root = if meta().exists("lambda_function_error") { throw("Invocation failed due to %v: %v".format(this.errorType, this.errorMessage)) } else { this } output: switch: retry_until_success: false cases: - check: errored() output: reject: ${! error() } - output: resource: somewhere_else ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/connect/guides/cloud/aws/). ### Common ```yml processor: label: "" aws_lambda: parallel: false function: "" # No default (required) ``` ### Advanced ```yml processor: label: "" aws_lambda: parallel: false function: "" # No default (required) rate_limit: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: false # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) timeout: 5s retries: 3 ``` ## [](#examples)Examples ### [](#branched-invoke)Branched Invoke This example uses a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) to map a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged. ```yaml pipeline: processors: - branch: request_map: '{"id":this.doc.id,"username":this.user.name}' processors: - aws_lambda: function: trigger_user_update ``` ## [](#fields)Fields ### [](#credentials-2)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#function)`function` The function to invoke. **Type**: `string` ### [](#parallel)`parallel` Whether messages of a batch should be dispatched in parallel. **Type**: `bool` **Default**: `false` ### [](#rate_limit)`rate_limit` An optional [`rate_limit`](https://docs.redpanda.com/connect/components/rate_limits/about/) to throttle invocations by. **Type**: `string` **Default**: `""` ### [](#region)`region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts for each message. **Type**: `int` **Default**: `3` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait before abandoning an invocation. **Type**: `string` **Default**: `5s` --- # Page 217: azure_cosmosdb **URL**: https://docs.redpanda.com/connect/components/processors/azure_cosmosdb.md --- # azure_cosmosdb --- title: azure_cosmosdb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/azure_cosmosdb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/azure_cosmosdb.adoc description: Creates or updates messages as JSON documents in Azure CosmosDB. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction). Introduced in version 4.25.0. When creating documents, each message must have the `id` property (case-sensitive) set (or use `auto_id: true`). It is the unique name that identifies the document, that is, no two documents share the same `id` within a logical partition. The `id` field must not exceed 255 characters. [See details](https://learn.microsoft.com/en-us/rest/api/cosmos-db/documents). The `partition_keys` field must resolve to the same value(s) across the entire message batch. ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#metadata)Metadata This component adds the following metadata fields to each message: - `activity_id` - `request_charge` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#batching)Batching CosmosDB limits the maximum batch size to 100 messages and the payload must not exceed 2MB ([details here](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-request-limits)). ### Common ```yml processor: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create item_id: "" # No default (optional) ``` ### Advanced ```yml processor: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create patch_operations: [] # No default (optional) patch_condition: "" # No default (optional) auto_id: true item_id: "" # No default (optional) enable_content_response_on_write: true ``` ## [](#examples)Examples ### [](#patch-documents)Patch documents Query documents from a container and patch them. ```yaml input: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = "AbyssalPlain" query: SELECT * FROM blobfish processors: - mapping: | root = "" meta habitat = json("habitat") meta id = this.id - azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: testdb container: blobfish partition_keys_map: root = json("habitat") item_id: ${! meta("id") } operation: Patch patch_operations: # Add a new /diet field - operation: Add path: /diet value_map: root = json("diet") # Remove the first location from the /locations array field - operation: Remove path: /locations/0 # Add new location at the end of the /locations array field - operation: Add path: /locations/- value_map: root = "Challenger Deep" # Return the updated document enable_content_response_on_write: true ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#auto_id)`auto_id` Automatically set the item `id` field to a random UUID v4. If the `id` field is already set, then it will not be overwritten. Setting this to `false` can improve performance, since the messages will not have to be parsed. **Type**: `bool` **Default**: `true` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#enable_content_response_on_write)`enable_content_response_on_write` Enable content response on write operations. To save some bandwidth, set this to false if you don’t need to receive the updated message(s) from the server, in which case the processor will not modify the content of the messages which are fed into it. Applies to every operation except Read. **Type**: `bool` **Default**: `true` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#item_id)`item_id` ID of item to replace or delete. Only used by the Replace and Delete operations This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: item_id: ${! json("id") } ``` ### [](#operation)`operation` Operation. **Type**: `string` **Default**: `Create` | Option | Summary | | --- | --- | | Create | Create operation. | | Delete | Delete operation. | | Patch | Patch operation. | | Read | Read operation. | | Replace | Replace operation. | | Upsert | Upsert operation. | ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = json("blobfish").depth ``` ### [](#patch_condition)`patch_condition` Patch operation condition. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: patch_condition: from c where not is_defined(c.blobfish) ``` ### [](#patch_operations)`patch_operations[]` Patch operations to be performed when `operation: Patch` . **Type**: `array` ### [](#patch_operations-operation)`patch_operations[].operation` Operation. **Type**: `string` **Default**: `Add` | Option | Summary | | --- | --- | | Add | Add patch operation. | | Increment | Increment patch operation. | | Remove | Remove patch operation. | | Replace | Replace patch operation. | | Set | Set patch operation. | ### [](#patch_operations-path)`patch_operations[].path` Path. **Type**: `string` ```yaml # Examples: path: /foo/bar/baz ``` ### [](#patch_operations-value_map)`patch_operations[].value_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to a value of any type that is supported by CosmosDB. **Type**: `string` ```yaml # Examples: value_map: root = "blobfish" # --- value_map: root = 41 # --- value_map: root = true # --- value_map: root = json("blobfish").depth # --- value_map: root = [1, 2, 3] ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 218: benchmark **URL**: https://docs.redpanda.com/connect/components/processors/benchmark.md --- # benchmark --- title: benchmark latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/benchmark page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/benchmark.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/benchmark.adoc description: Logs basic throughput statistics of messages that pass through this processor. page-git-created-date: "2024-12-16" page-git-modified-date: "2026-10-10" --- Logs throughput statistics for processed messages, and provides a summary of those statistics over the lifetime of the processor. Introduced in version 4.40.0. #### Common ```yml processor: label: "" benchmark: interval: 5s count_bytes: true ``` #### Advanced ```yml processor: label: "" benchmark: interval: 5s count_bytes: true ``` ## [](#throughput-statistics)Throughput statistics This processor logs the following rolling statistics at a [configurable interval](#interval) to help you to understand the current performance of your pipeline: - The number of messages processed per second. - The number of bytes processed per second (optional). For example: ```bash INFO rolling stats: 1 msg/sec, 407 B/sec ``` When the processor shuts down, it also logs a summary of the number and size of messages processed during its lifetime. For example: ```bash INFO total stats: 1.00186 msg/sec, 425 B/sec ``` ## [](#fields)Fields ### [](#count_bytes)`count_bytes` Whether to measure the number of bytes per second of throughput. If set to `true`, structured data must be serialized to count the number of bytes processed, which can unnecessarily degrade performance if serialization is not required elsewhere in your pipeline. **Type**: `bool` **Default**: `true` ### [](#interval)`interval` How often to emit rolling statistics. Set to `0` to log only summary statistics when the processor shuts down. **Type**: `string` **Default**: `5s` --- # Page 219: bloblang **URL**: https://docs.redpanda.com/connect/components/processors/bloblang.md --- # bloblang --- title: bloblang latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/bloblang page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/bloblang.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/bloblang.adoc description: Executes a Bloblang mapping on messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-14" --- Executes a [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) mapping on messages. Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information see [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. ## [](#component-rename)Component rename This processor was recently renamed to the [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) in order to make the purpose of the processor more prominent. It is still valid to use the existing `bloblang` name but eventually it will be deprecated and replaced by the new name in example configs. ### Common ```yml processor: label: "" bloblang: "" ``` ### Advanced ```yml processor: label: "" bloblang: "" ``` ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the fans to only those with an obsession score above 0.5, giving us: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - bloblang: | root = this root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - bloblang: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired fallback behavior, which you can read about in [Error handling](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). --- # Page 220: bounds_check **URL**: https://docs.redpanda.com/connect/components/processors/bounds_check.md --- # bounds_check --- title: bounds_check latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/bounds_check page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/bounds_check.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/bounds_check.adoc description: Removes messages (and batches) that do not fit within certain size boundaries. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Removes messages (and batches) that do not fit within certain size boundaries. #### Common ```yml processor: label: "" bounds_check: max_part_size: 1073741824 min_part_size: 1 ``` #### Advanced ```yml processor: label: "" bounds_check: max_part_size: 1073741824 min_part_size: 1 max_parts: 100 min_parts: 1 ``` ## [](#fields)Fields ### [](#max_part_size)`max_part_size` The maximum size of a message to allow (in bytes) **Type**: `int` **Default**: `1073741824` ### [](#max_parts)`max_parts` The maximum size of message batches to allow (in message count) **Type**: `int` **Default**: `100` ### [](#min_part_size)`min_part_size` The minimum size of a message to allow (in bytes) **Type**: `int` **Default**: `1` ### [](#min_parts)`min_parts` The minimum size of message batches to allow (in message count) **Type**: `int` **Default**: `1` --- # Page 221: branch **URL**: https://docs.redpanda.com/connect/components/processors/branch.md --- # branch --- title: branch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/branch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/branch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/branch.adoc description: The branch processor allows you to create a new request message via a Bloblang mapping, execute a list of processors on the request messages, and, finally, map the result back into the source message using another mapping. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- The `branch` processor allows you to create a new request message via a [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/), execute a list of processors on the request messages, and, finally, map the result back into the source message using another mapping. This is useful for preserving the original message contents when using processors that would otherwise replace the entire contents. ## [](#metadata)Metadata Metadata fields that are added to messages during branch processing will not be automatically copied into the resulting message. In order to do this you should explicitly declare in your `result_map` either a wholesale copy with `meta = metadata()`, or selective copies with `meta foo = metadata("bar")` and so on. It is also possible to reference the metadata of the origin message in the `result_map` using the [`@` operator](https://docs.redpanda.com/connect/guides/bloblang/about/#metadata). ## [](#error-handling)Error handling If the `request_map` fails the child processors will not be executed. If the child processors themselves result in an (uncaught) error then the `result_map` will not be executed. If the `result_map` fails the message will remain unchanged. Under any of these conditions standard [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/) can be used in order to filter, DLQ or recover the failed messages. ## [](#conditional-branching)Conditional branching If the root of your request map is set to `deleted()` then the branch processors are skipped for the given message, this allows you to conditionally branch messages. ### Common ```yml processor: label: "" branch: request_map: "" processors: [] # No default (required) result_map: "" ``` ### Advanced ```yml processor: label: "" branch: request_map: "" processors: [] # No default (required) result_map: "" ``` ## [](#fields)Fields ### [](#processors)`processors[]` A list of processors to apply to mapped requests. When processing message batches the resulting batch must match the size and ordering of the input batch, therefore filtering, grouping should not be performed within these processors. **Type**: `array` ### [](#request_map)`request_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that describes how to create a request payload suitable for the child processors of this branch. If left empty then the branch will begin with an exact copy of the origin message (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: request_map: |- root = { "id": this.doc.id, "content": this.doc.body.text } # --- request_map: |- root = if this.type == "foo" { this.foo.request } else { deleted() } ``` ### [](#result_map)`result_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that describes how the resulting messages from branched processing should be mapped back into the original payload. If left empty the origin message will remain unchanged (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: result_map: |- meta foo_code = metadata("code") root.foo_result = this # --- result_map: |- meta = metadata() root.bar.body = this.body root.bar.id = this.user.id # --- result_map: root.raw_result = content().string() # --- result_map: |- root.enrichments.foo = if metadata("request_failed") != null { throw(metadata("request_failed")) } else { this } # --- result_map: |- # Retain only the updated metadata fields which were present in the origin message meta = metadata().filter(v -> @.get(v.key) != null) ``` ## [](#examples)Examples ### [](#http-request)HTTP Request This example strips the request message into an empty body, grabs an HTTP payload, and places the result back into the original message at the path `image.pull_count`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: root.image.pull_count = this.pull_count # Example input: {"id":"foo","some":"pre-existing data"} # Example output: {"id":"foo","some":"pre-existing data","image":{"pull_count":1234}} ``` ### [](#non-structured-results)Non Structured Results When the result of your branch processors is unstructured and you wish to simply set a resulting field to the raw output use the content function to obtain the raw bytes of the resulting message and then coerce it into your value type of choice: ```yaml pipeline: processors: - branch: request_map: 'root = this.document.id' processors: - cache: resource: descriptions_cache key: ${! content() } operator: get result_map: root.document.description = content().string() # Example input: {"document":{"id":"foo","content":"hello world"}} # Example output: {"document":{"id":"foo","content":"hello world","description":"this is a cool doc"}} ``` ### [](#lambda-function)Lambda Function This example maps a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged. ```yaml pipeline: processors: - branch: request_map: '{"id":this.doc.id,"username":this.user.name}' processors: - aws_lambda: function: trigger_user_update # Example input: {"doc":{"id":"foo","body":"hello world"},"user":{"name":"fooey"}} # Output matches the input, which is unchanged ``` ### [](#conditional-caching)Conditional Caching This example caches a document by a message ID only when the type of the document is a foo: ```yaml pipeline: processors: - branch: request_map: | meta id = this.id root = if this.type == "foo" { this.document } else { deleted() } processors: - cache: resource: TODO operator: set key: ${! @id } value: ${! content() } ``` --- # Page 222: cache **URL**: https://docs.redpanda.com/connect/components/processors/cache.md --- # cache --- title: cache latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/cache page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cache.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cache.adoc description: Performs operations against a cache resource for each message, allowing you to store or retrieve data within message payloads. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs operations against a [cache resource](https://docs.redpanda.com/connect/components/caches/about/) for each message, allowing you to store or retrieve data within message payloads. For use cases where you wish to cache the result of processors consider using the [`cached` processor](https://docs.redpanda.com/connect/components/processors/cached/) instead. This processor will interpolate functions within the `key` and `value` fields individually for each message. This allows you to specify dynamic keys and values based on the contents of the message payloads and metadata. You can find a list of functions in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). #### Common ```yml processor: label: "" cache: resource: "" # No default (required) operator: "" # No default (required) key: "" # No default (required) value: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" cache: resource: "" # No default (required) operator: "" # No default (required) key: "" # No default (required) value: "" # No default (optional) ttl: "" # No default (optional) ``` ## [](#examples)Examples ### [](#deduplication)Deduplication Deduplication can be done using the add operator with a key extracted from the message payload, since it fails when a key already exists we can remove the duplicates using a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/): ```yaml pipeline: processors: - cache: resource: foocache operator: add key: '${! json("message.id") }' value: "storeme" - mapping: root = if errored() { deleted() } cache_resources: - label: foocache redis: url: tcp://TODO:6379 ``` ### [](#deduplication-batch-wide)Deduplication Batch-Wide Sometimes it’s necessary to deduplicate a batch of messages (also known as a window) by a single identifying value. This can be done by introducing a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/), which executes the cache only once on behalf of the batch, in this case with a value make from a field extracted from the first and last messages of the batch: ```yaml pipeline: processors: # Try and add one message to a cache that identifies the whole batch - branch: request_map: | root = if batch_index() == 0 { json("id").from(0) + json("meta.tail_id").from(-1) } else { deleted() } processors: - cache: resource: foocache operator: add key: ${! content() } value: t # Delete all messages if we failed - mapping: | root = if errored().from(0) { deleted() } ``` ### [](#hydration)Hydration It’s possible to enrich payloads with content previously stored in a cache by using the [`branch`](https://docs.redpanda.com/connect/components/processors/branch/) processor: ```yaml pipeline: processors: - branch: processors: - cache: resource: foocache operator: get key: '${! json("message.document_id") }' result_map: 'root.message.document = this' # NOTE: If the data stored in the cache is not valid JSON then use # something like this instead: # result_map: 'root.message.document = content().string()' cache_resources: - label: foocache memcached: addresses: [ "TODO:11211" ] ``` ## [](#fields)Fields ### [](#key)`key` A key to use with the cache. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#operator)`operator` The [operation](#operators) to perform with the cache. **Type**: `string` **Options**: `set`, `add`, `get`, `delete`, `exists` ### [](#resource)`resource` The [`cache` resource](https://docs.redpanda.com/connect/components/caches/about/) to target with this processor. **Type**: `string` ### [](#ttl)`ttl` The time to live (TTL) of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, those that do will have a configuration field `default_ttl`, and those that do not will fall back to their generally configured TTL setting. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 3.33.0 or later. **Type**: `string` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` ### [](#value)`value` A value to use with the cache (when applicable). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ## [](#operators)Operators ### [](#set)`set` Set a key in the cache to a value. If the key already exists the contents are overridden. ### [](#add)`add` Set a key in the cache to a value. If the key already exists the action fails with a 'key already exists' error, which can be detected with [processor error handling](https://docs.redpanda.com/connect/configuration/error_handling/). ### [](#get)`get` Retrieve the contents of a cached key and replace the original message payload with the result. If the key does not exist the action fails with an error, which can be detected with [processor error handling](https://docs.redpanda.com/connect/configuration/error_handling/). ### [](#exists)`exists` Check whether a specific key is in the cache and replace the original message payload with `true` if the key exists, or `false` if it doesn’t. ### [](#delete)`delete` Delete a key and its contents from the cache. If the key does not exist the action is a no-op and will not fail with an error. --- # Page 223: cached **URL**: https://docs.redpanda.com/connect/components/processors/cached.md --- # cached --- title: cached latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/cached page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cached.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cached.adoc description: Cache the result of applying one or more processors to messages identified by a key. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Cache the result of applying one or more processors to messages identified by a key. If the key already exists within the cache the contents of the message will be replaced with the cached result instead of applying the processors. This component is therefore useful in situations where an expensive set of processors need only be executed periodically. Introduced in version 4.3.0. The format of the data when stored within the cache is a custom and versioned schema chosen to balance performance and storage space. It is therefore not possible to point this processor to a cache that is pre-populated with data that this processor has not created itself. #### Common ```yml processor: label: "" cached: cache: "" # No default (required) skip_on: "" # No default (optional) key: "" # No default (required) ttl: "" # No default (optional) processors: [] # No default (required) ``` #### Advanced ```yml processor: label: "" cached: cache: "" # No default (required) skip_on: "" # No default (optional) key: "" # No default (required) ttl: "" # No default (optional) processors: [] # No default (required) ``` ## [](#examples)Examples ### [](#cached-enrichment)Cached Enrichment In the following example we want to we enrich messages consumed from Kafka with data specific to the origin topic partition, we do this by placing an `http` processor within a `branch`, where the HTTP URL contains interpolation functions with the topic and partition in the path. However, it would be inefficient to make this HTTP request for every single message as the result is consistent for all data of a given topic partition. We can solve this by placing our enrichment call within a `cached` processor where the key contains the topic and partition, resulting in messages that originate from the same topic/partition combination using the cached result of the prior. ```yaml pipeline: processors: - branch: processors: - cached: key: '${! meta("kafka_topic") }-${! meta("kafka_partition") }' cache: foo_cache processors: - mapping: 'root = ""' - http: url: http://example.com/enrichment/${! meta("kafka_topic") }/${! meta("kafka_partition") } verb: GET result_map: 'root.enrichment = this' cache_resources: - label: foo_cache memory: # Disable compaction so that cached items never expire compaction_interval: "" ``` ### [](#periodic-global-enrichment)Periodic Global Enrichment In the following example we enrich all messages with the same data obtained from a static URL with an `http` processor within a `branch`. However, we expect the data from this URL to change roughly every 10 minutes, so we configure a `cached` processor with a static key (since this request is consistent for all messages) and a TTL of `10m`. ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - cached: key: static_foo cache: foo_cache ttl: 10m processors: - http: url: http://example.com/get/foo.json verb: GET result_map: 'root.foo = this' cache_resources: - label: foo_cache memory: {} ``` ## [](#fields)Fields ### [](#cache)`cache` The cache resource to read and write processor results from. **Type**: `string` ### [](#key)`key` A key to be resolved for each message, if the key already exists in the cache then the cached result is used, otherwise the processors are applied and the result is cached under this key. The key could be static and therefore apply generally to all messages or it could be an interpolated expression that is potentially unique for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: my_foo_result # --- key: ${! this.document.id } # --- key: ${! meta("kafka_key") } # --- key: ${! meta("kafka_topic") } ``` ### [](#processors)`processors[]` The list of processors whose result will be cached. **Type**: `array` ### [](#skip_on)`skip_on` A condition that can be used to skip caching the results from the processors. Requires version 4.16.0 or later. **Type**: `string` ```yaml # Examples: skip_on: errored() ``` ### [](#ttl)`ttl` An optional expiry period to set for each cache entry. Some caches only have a general TTL and will therefore ignore this setting. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 224: catch **URL**: https://docs.redpanda.com/connect/components/processors/catch.md --- # catch --- title: catch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/catch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/catch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/catch.adoc description: Applies a list of child processors _only_ when a previous processing step has failed. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Applies a list of child processors _only_ when a previous processing step has failed. Behaves similarly to the [`for_each`](https://docs.redpanda.com/connect/components/processors/for_each/) processor, where a list of child processors are applied to individual messages of a batch. However, processors are only applied to messages that failed a processing step prior to the catch. For example, with the following config: ```yaml pipeline: processors: - resource: foo - catch: - resource: bar - resource: baz ``` If the processor `foo` fails for a particular message, that message will be fed into the processors `bar` and `baz`. Messages that do not fail for the processor `foo` will skip these processors. When messages leave the catch block their fail flags are cleared. This processor is useful for when it’s possible to recover failed messages, or when special actions (such as logging/metrics) are required before dropping them. > 📝 **NOTE** > > When strict error handling (`error_handling.strict`) is enabled this processor does not recover anything, as a failed message short-circuits past it rather than reaching it. Use the [`try_catch`](https://docs.redpanda.com/connect/components/processors/try_catch/) processor instead, which contains both the fallible step and its recovery. More information about error handling can be found in [Error Handling](https://docs.redpanda.com/connect/configuration/error_handling/). ## Common ```yml processor: label: "" catch: [] ``` ## Advanced ```yml processor: label: "" catch: [] ``` --- # Page 225: cohere_chat **URL**: https://docs.redpanda.com/connect/components/processors/cohere_chat.md --- # cohere_chat --- title: cohere_chat latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/cohere_chat page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cohere_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cohere_chat.adoc description: Generates responses to messages in a chat conversation, using the Cohere API. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Generates responses to messages in a chat conversation, using the [Cohere API](https://docs.cohere.com/docs/chat-api) and external tools. Introduced in version 4.37.0. ### Common ```yml processor: label: "" cohere_chat: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) response_format: text json_schema: "" # No default (optional) max_tool_calls: 10 tools: [] ``` ### Advanced ```yml processor: label: "" cohere_chat: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) response_format: text json_schema: "" # No default (optional) schema_registry: url: "" # No default (required) subject: "" # No default (required) refresh_interval: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} top_p: 0 # No default (optional) frequency_penalty: 0 # No default (optional) presence_penalty: 0 # No default (optional) seed: 0 # No default (optional) stop: [] # No default (optional) max_tool_calls: 10 tools: [] ``` This processor sends the contents of user prompts to the Cohere API, which generates responses using all available context, including supplementary data provided by external tools. By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` field to customize it. To learn more about chat completion, see the [Cohere API documentation](https://docs.cohere.com/docs/chat-api). ## [](#fields)Fields ### [](#api_key)`api_key` Your API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#frequency_penalty)`frequency_penalty` A number between `0.0` and `1.0`. Higher values penalize new tokens more strongly based on the frequency of their appearance in the prompt and the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#json_schema)`json_schema` The JSON schema to use when responding in `json_schema` format. To learn more about the JSON schema features supported, see the [Cohere documentation](https://docs.cohere.com/docs/structured-outputs-json). **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to allow in the chat completion. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of tool calls the model can perform. Requires version 4.53.0 or later. **Type**: `int` **Default**: `10` ### [](#model)`model` The name of the Cohere model you want to use. **Type**: `string` ```yaml # Examples: model: command-r-plus # --- model: command-r # --- model: command # --- model: command-light ``` ### [](#presence_penalty)`presence_penalty` A number between `0.0` and `1.0`. Higher values penalize new tokens more strongly if they have already appeared in the prompt or the text so far, regardless of how often. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` Choose the model’s output format. If `json_schema` is specified, then you must also configure a `json_schema` or `schema_registry`. **Type**: `string` **Default**: `text` **Options**: `text`, `json`, `json_schema` ### [](#schema_registry)`schema_registry` The schema registry to dynamically load schemas from when responding in `json_schema` format. Schemas themselves must be in JSON format. To learn more about the JSON schema features supported, see the [Cohere documentation](https://docs.cohere.com/docs/structured-outputs-json). **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-refresh_interval)`schema_registry.refresh_interval` The refresh rate for fetching the latest schema. If not specified the schema does not refresh. **Type**: `string` ### [](#schema_registry-subject)`schema_registry.subject` The subject name to fetch the schema for. **Type**: `string` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed)`seed` If specified, the Cohere API makes a best effort to sample deterministically. Repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed. **Type**: `int` ### [](#stop)`stop[]` Specify up to five sequences to stop the API from generating further tokens. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` The sampling temperature, between `0` and `2`, that tunes the randomness of the generated text. Lower values make the output more focused and deterministic, and higher values make it more random. If you don’t set this field, the Cohere API uses its default of `0.3`. To increase randomness further, also raise `top_p`. **Type**: `float` ### [](#tools)`tools[]` External tools that the model can invoke, such as functions, APIs, or web browsing. You can define subpipelines of processors that implement these tools, enabling the model to use agent-like behavior to decide when and how to invoke them to enhance response generation. Requires version 4.53.0 or later. **Type**: `array` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool. The LLM uses it to decide whether to invoke the tool. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The parameters the LLM can provide when it invokes this tool, keyed by parameter name. **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` The values this parameter is limited to. Leave empty to accept any value. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The names of the parameters the LLM must provide when it invokes this tool. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The processors to run when the LLM invokes this tool. They receive a message whose payload is the tool call arguments as a JSON object, and their output is returned to the LLM as the tool result. **Type**: `array` ### [](#top_p)`top_p` Nucleus sampling, sent to the Cohere API as the `p` parameter. The model considers only the most likely tokens whose total probability mass is `top_p`. The Cohere API accepts values from `0.01` to `0.99`, and uses `0.75` if you don’t set this field. **Type**: `float` ## [](#example)Example In this pipeline configuration, the Command R+ model executes a number of processors, which make a tool call to retrieve weather data for a specific city. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - cohere_chat: auth_token: my_cohere_api_token model: command-r-plus prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to look up the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: User-Agent: curl/8.11.1 # Returns a text string from the weather website output: stdout: {} ``` --- # Page 226: cohere_embeddings **URL**: https://docs.redpanda.com/connect/components/processors/cohere_embeddings.md --- # cohere_embeddings --- title: cohere_embeddings latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/cohere_embeddings page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cohere_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cohere_embeddings.adoc description: Generates vector embeddings to represent input text, using the Cohere API. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Generates vector embeddings to represent input text, using the [Cohere API](https://docs.cohere.com/docs/embeddings). Introduced in version 4.37.0. #### Common ```yml processor: label: "" cohere_embeddings: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) text_mapping: "" # No default (optional) input_type: search_document dimensions: 0 # No default (optional) ``` #### Advanced ```yml processor: label: "" cohere_embeddings: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) text_mapping: "" # No default (optional) input_type: search_document dimensions: 0 # No default (optional) ``` This processor sends text strings to your chosen large language model (LLM), which generates vector embeddings for them using the Cohere API. By default, the processor submits the entire payload of each message as a string, unless you use the `text_mapping` field to customize it. To learn more about vector embeddings, see the [Cohere API documentation](https://docs.cohere.com/docs/embeddings). ## [](#examples)Examples ### [](#store-embedding-vectors-in-qdrant)Store embedding vectors in Qdrant Computes embeddings for some generated data and stores them in [Qdrant](https://docs.redpanda.com/connect/components/outputs/qdrant/). ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - cohere_embeddings: model: embed-english-v3 api_key: "${COHERE_API_KEY}" text_mapping: "root = this.text" output: qdrant: grpc_host: localhost:6334 collection_name: "example_collection" id: "root = uuid_v4()" vector_mapping: "root = this" ``` ## [](#fields)Fields ### [](#api_key)`api_key` Your API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#dimensions)`dimensions` The number of dimensions (numerical values) in each vector embedding generated by this processor. This parameter only supports [`embed-v4.0`](https://docs.cohere.com/v2/docs/embeddings) and newer models. Possible values are `256`, `512`, `1024`, and `1536`. **Type**: `int` ### [](#input_type)`input_type` The type of text input passed to the model. Requires version 4.53.0 or later. **Type**: `string` **Default**: `search_document` | Option | Summary | | --- | --- | | classification | Used for embeddings passed through a text classifier. | | clustering | Used for the embeddings run through a clustering algorithm. | | search_document | Used for embeddings stored in a vector database for search use-cases. | | search_query | Used for embeddings of search queries run against a vector DB to find relevant documents. | ### [](#model)`model` The name of the Cohere model you want to use. **Type**: `string` ```yaml # Examples: model: embed-english-v3.0 # --- model: embed-english-light-v3.0 # --- model: embed-multilingual-v3.0 # --- model: embed-multilingual-light-v3.0 ``` ### [](#text_mapping)`text_mapping` A Bloblang mapping that returns the text you want to generate vector embeddings for. By default, the processor submits the entire payload of each message as a string. **Type**: `string` --- # Page 227: cohere_rerank **URL**: https://docs.redpanda.com/connect/components/processors/cohere_rerank.md --- # cohere_rerank --- title: cohere_rerank latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/cohere_rerank page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cohere_rerank.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cohere_rerank.adoc description: Generates vector embeddings to represent input text, using the Cohere API. page-git-created-date: "2025-05-19" page-git-modified-date: "2026-10-10" --- Sends document strings to the [Cohere API](https://docs.cohere.com/reference/rerank), which returns them [ranked by their relevance to a specified query](https://docs.cohere.com/docs/rerank-2). The output of this processor is an array of strings, ordered by their relevance to the query. Introduced in version 4.53.0. #### Common ```yml processor: label: "" cohere_rerank: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) query: "" # No default (required) documents: "" # No default (required) top_n: "0" max_tokens_per_doc: 4096 ``` #### Advanced ```yml processor: label: "" cohere_rerank: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) query: "" # No default (required) documents: "" # No default (required) top_n: "0" max_tokens_per_doc: 4096 ``` ## [](#metadata)Metadata - `relevance_scores`: An array of scores for each input document that indicates how relevant it is to the query. The scores are in the same order as the documents in the input. The higher the score, the more relevant the document. ## [](#examples)Examples ### [](#rerank-some-documents-based-on-a-query)Rerank some documents based on a query Rerank some documents based on a query ```yaml input: generate: interval: 1s mapping: | root = { "query": fake("sentence"), "docs": [fake("paragraph"), fake("paragraph"), fake("paragraph")], } pipeline: processors: - cohere_rerank: model: rerank-v3.5 api_key: "${COHERE_API_KEY}" query: "${!this.query}" documents: "root = this.docs" output: stdout: {} ``` ## [](#fields)Fields ### [](#api_key)`api_key` Your API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#documents)`documents` A list of text strings that are compared to the specified query. For optimal performance, Cohere recommends that you: - Send fewer than 1000 documents in a single request. - Send structured data in YAML format. **Type**: `string` ### [](#max_tokens_per_doc)`max_tokens_per_doc` This processor automatically truncates long documents to the specified number of tokens. **Type**: `int` **Default**: `4096` ### [](#model)`model` The name of the Cohere model you want to use. **Type**: `string` ```yaml # Examples: model: rerank-v3.5 ``` ### [](#query)`query` The search query you want to execute. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#top_n)`top_n` The number of documents to return when the query is executed. If set to `0`, all documents are returned. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `0` --- # Page 228: command **URL**: https://docs.redpanda.com/connect/components/processors/command.md --- # command --- title: command latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/command page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/command.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/command.adoc description: Executes a command for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a command for each message. Introduced in version 4.21.0. #### Common ```yml processor: label: "" command: name: "" # No default (required) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" command: name: "" # No default (required) args_mapping: "" # No default (optional) ``` The specified command is executed for each message processed, with the raw bytes of the message being fed into the stdin of the command process, and the resulting message having its contents replaced with the stdout of it. ## [](#performance)Performance Since this processor executes a new process for each message performance will likely be an issue for high throughput streams. If this is the case then consider using the [`subprocess` processor](https://docs.redpanda.com/connect/components/processors/subprocess/) instead as it keeps the underlying process alive long term and uses codecs to insert and extract inputs and outputs to it via stdin/stdout. ## [](#error-handling)Error handling If a non-zero error code is returned by the command then an error containing the entirety of stderr (or a generic message if nothing is written) is set on the message. These failed messages will continue through the pipeline unchanged, but can be dropped or placed in a dead letter queue according to your config, you can read about [these patterns](https://docs.redpanda.com/connect/configuration/error_handling/). If the command is successful but stderr is written to then a metadata field `command_stderr` is populated with its contents. ## [](#metadata)Metadata This processor adds the following metadata fields to each message: - `command_stderr` - Contains the stderr output of a successful command, if any. - `exit_code` - The exit code returned by the command. You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that, when specified, should resolve into an array of arguments to pass to the command. Command arguments are expressed this way in order to support dynamic behavior. **Type**: `string` ```yaml # Examples: args_mapping: '[ "-c", this.script_path ]' ``` ### [](#name)`name` The name of the command to execute. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: name: bash # --- name: go # --- name: ${! @command } ``` ## [](#examples)Examples ### [](#cron-scheduled-command)Cron Scheduled Command This example uses a [`generate` input](https://docs.redpanda.com/connect/components/inputs/generate/) to trigger a command on a cron schedule: ```yaml input: generate: interval: '0,30 */2 * * * *' mapping: 'root = ""' # Empty string as we do not need to pipe anything to stdin processors: - command: name: df args_mapping: '[ "-h" ]' ``` ### [](#dynamic-command-execution)Dynamic Command Execution This example config takes structured messages of the form `{"command":"echo","args":["foo"]}` and uses their contents to execute the contained command and arguments dynamically, replacing its contents with the command result printed to stdout: ```yaml pipeline: processors: - command: name: ${! this.command } args_mapping: 'this.args' ``` --- # Page 229: compress **URL**: https://docs.redpanda.com/connect/components/processors/compress.md --- # compress --- title: compress latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/compress page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/compress.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/compress.adoc description: "Compresses messages according to the selected algorithm. Supported compression algorithms are: [flate gzip lz4 pgzip snappy zlib]." page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Compresses messages according to the selected algorithm. Supported compression algorithms are: \[flate gzip lz4 pgzip snappy zlib\] The 'level' field might not apply to all algorithms. #### Common ```yml processor: label: "" compress: algorithm: "" # No default (required) level: -1 ``` #### Advanced ```yml processor: label: "" compress: algorithm: "" # No default (required) level: -1 ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` The compression algorithm to use. **Type**: `string` **Options**: `flate`, `gzip`, `lz4`, `pgzip`, `snappy`, `zlib` ### [](#level)`level` The level of compression to use. May not be applicable to all algorithms. **Type**: `int` **Default**: `-1` --- # Page 230: couchbase **URL**: https://docs.redpanda.com/connect/components/processors/couchbase.md --- # couchbase --- title: couchbase latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/couchbase page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/couchbase.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/couchbase.adoc description: Performs operations against Couchbase for each message, allowing you to store or retrieve data within message payloads. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs operations against Couchbase for each message, allowing you to store or retrieve data within message payloads. Introduced in version 4.11.0. When inserting, replacing or upserting documents, each must have the `content` property set. #### Common ```yml processor: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) id: "" # No default (required) content: "" # No default (optional) operation: get ``` #### Advanced ```yml processor: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) collection: "" # No default (optional) scope: "" # No default (optional) transcoder: legacy timeout: 15s id: "" # No default (required) content: "" # No default (optional) ttl: "" # No default (optional) operation: get ``` ## [](#fields)Fields ### [](#bucket)`bucket` The name of the Couchbase bucket to access. **Type**: `string` ### [](#collection)`collection` The name of the collection within the bucket to access. **Type**: `string` ### [](#content)`content` Document content. **Type**: `string` ### [](#id)`id` Document id. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${! json("id") } ``` ### [](#operation)`operation` Couchbase operation to perform. **Type**: `string` **Default**: `get` | Option | Summary | | --- | --- | | get | fetch a document. | | insert | insert a new document. | | remove | delete a document. | | replace | replace the contents of a document. | | upsert | creates a new document if it does not exist, if it does exist then it updates it. | ### [](#password)`password` A password to authenticate with the Couchbase cluster. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#scope)`scope` Bucket scope. Requires version 4.56.0 or later. **Type**: `string` ### [](#timeout)`timeout` The timeout for Couchbase operations, including connecting to the cluster and waiting for the bucket to become ready. **Type**: `string` **Default**: `15s` ### [](#transcoder)`transcoder` The Couchbase transcoder to use for converting the data format of documents written to and read from Couchbase. **Type**: `string` **Default**: `legacy` | Option | Summary | | --- | --- | | json | JSONTranscoder implements the default transcoding behavior and applies JSON transcoding to all values. This will apply the following behavior to the value: binary ([]byte) → error. default → JSON value, JSON Flags. | | legacy | LegacyTranscoder implements the behavior for a backward-compatible transcoder. This transcoder implements behavior matching that of gocb v1.This will apply the following behavior to the value: binary ([]byte) → binary bytes, Binary expectedFlags. string → string bytes, String expectedFlags. default → JSON value, JSON expectedFlags. | | raw | RawBinaryTranscoder implements passthrough behavior of raw binary data. This transcoder does not apply any serialization. This will apply the following behavior to the value: binary ([]byte) → binary bytes, binary expectedFlags. default → error. | | rawjson | RawJSONTranscoder implements passthrough behavior of JSON data. This transcoder does not apply any serialization. It will forward data across the network without incurring unnecessary parsing costs. This will apply the following behavior to the value: binary ([]byte) → JSON bytes, JSON expectedFlags. string → JSON bytes, JSON expectedFlags. default → error. | | rawstring | RawStringTranscoder implements passthrough behavior of raw string data. This transcoder does not apply any serialization. This will apply the following behavior to the value: string → string bytes, string expectedFlags. default → error. | ### [](#ttl)`ttl` An optional TTL to set for items. Requires version 4.73.0 or later. **Type**: `string` ### [](#url)`url` The Couchbase connection string. **Type**: `string` ```yaml # Examples: url: couchbase://localhost:11210 ``` ### [](#username)`username` A username to authenticate with the Couchbase cluster. **Type**: `string` --- # Page 231: crash **URL**: https://docs.redpanda.com/connect/components/processors/crash.md --- # crash --- title: crash latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/crash page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/crash.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/crash.adoc description: Crashes the process using a fatal log message. The log message can be set using function interpolations described in Bloblang queries which allows you to log the contents and metadata of messages. page-git-created-date: "2025-03-11" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is in beta. Its behavior might change in a future release. Crashes the process using a fatal log message. The log message can be set using function interpolations described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) which allows you to log the contents and metadata of messages. ## Common ```yml processor: label: "" crash: "" # No default (required) ``` ## Advanced ```yml processor: label: "" crash: "" # No default (required) ``` This processor is useful for detecting unhandled errors during development and testing. For more information about common patterns for trapping errors, see [Error Handling](https://docs.redpanda.com/connect/configuration/error_handling/). The value of `crash` is the fatal log message, and supports [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). For example: ```yaml pipeline: processors: - crash: "Processor ${!error_source_label()} failed due to: ${!error()}" ``` --- # Page 232: decompress **URL**: https://docs.redpanda.com/connect/components/processors/decompress.md --- # decompress --- title: decompress latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/decompress page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/decompress.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/decompress.adoc description: "Decompresses messages according to the selected algorithm. Supported decompression algorithms are: [bzip2 flate gzip lz4 pgzip snappy zlib]." page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Decompresses messages according to the selected algorithm. Supported decompression algorithms are: \[bzip2 flate gzip lz4 pgzip snappy zlib\] #### Common ```yml processor: label: "" decompress: algorithm: "" # No default (required) ``` #### Advanced ```yml processor: label: "" decompress: algorithm: "" # No default (required) ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` The decompression algorithm to use. **Type**: `string` **Options**: `bzip2`, `flate`, `gzip`, `lz4`, `pgzip`, `snappy`, `zlib` --- # Page 233: dedupe **URL**: https://docs.redpanda.com/connect/components/processors/dedupe.md --- # dedupe --- title: dedupe latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/dedupe page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/dedupe.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/dedupe.adoc description: Deduplicates messages by storing a key value in a cache using the add operator. If the key already exists within the cache it is dropped. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Deduplicates messages by storing a key value in a cache using the `add` operator. If the key already exists within the cache it is dropped. Caches must be configured as resources, for more information check out the [cache documentation](https://docs.redpanda.com/connect/components/caches/about/). When using this processor with an output target that might fail you should always wrap the output within an indefinite [`retry`](https://docs.redpanda.com/connect/components/outputs/retry/) block. This ensures that during outages your messages aren’t reprocessed after failures, which would result in messages being dropped. ## [](#batch-deduplication)Batch deduplication This processor enacts on individual messages only, in order to perform a deduplication on behalf of a batch (or window) of messages instead use the [`cache` processor](https://docs.redpanda.com/connect/components/processors/cache/#examples). ## [](#delivery-guarantees)Delivery guarantees Performing deduplication on a stream using a distributed cache voids any at-least-once guarantees that it previously had. This is because the cache will preserve message signatures even if the message fails to leave the Redpanda Connect pipeline, which would cause message loss in the event of an outage at the output sink followed by a restart of the Redpanda Connect instance (or a server crash, etc). This problem can be mitigated by using an in-memory cache and distributing messages to horizontally scaled Redpanda Connect pipelines partitioned by the deduplication key. However, in situations where at-least-once delivery guarantees are important it is worth avoiding deduplication in favour of implement idempotent behavior at the edge of your stream pipelines. ### Common ```yml processor: label: "" dedupe: cache: "" # No default (required) key: "" # No default (required) drop_on_err: true ``` ### Advanced ```yml processor: label: "" dedupe: cache: "" # No default (required) key: "" # No default (required) drop_on_err: true ``` ## [](#fields)Fields ### [](#cache)`cache` The [`cache` resource](https://docs.redpanda.com/connect/components/caches/about/) to target with this processor. **Type**: `string` ### [](#drop_on_err)`drop_on_err` Whether messages should be dropped when the cache returns a general error such as a network issue. **Type**: `bool` **Default**: `true` ### [](#key)`key` An interpolated string yielding the key to deduplicate by for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: ${! meta("kafka_key") } # --- key: ${! content().hash("xxhash64") } ``` ## [](#examples)Examples ### [](#deduplicate-based-on-kafka-key)Deduplicate based on Kafka key The following configuration demonstrates a pipeline that deduplicates messages based on the Kafka key. ```yaml pipeline: processors: - dedupe: cache: keycache key: ${! meta("kafka_key") } cache_resources: - label: keycache memory: default_ttl: 60s ``` --- # Page 234: ffi **URL**: https://docs.redpanda.com/connect/components/processors/ffi.md --- # ffi --- title: ffi latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/ffi page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ffi.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ffi.adoc description: Invoke a function within a shared library as a processing step. page-git-created-date: "2026-01-05" page-git-modified-date: "2026-10-10" --- Invoke a function within a shared library as a processing step. #### Common ```yml processor: label: "" ffi: library_path: "" # No default (required) function_name: "" # No default (required) args_mapping: "" # No default (required) signature: return: type: "" # No default (required) parameters: [] # No default (required) ``` #### Advanced ```yml processor: label: "" ffi: library_path: "" # No default (required) function_name: "" # No default (required) args_mapping: "" # No default (required) signature: return: type: "" # No default (required) parameters: [] # No default (required) ``` The `ffi` processor allows you to dynamically load shared libraries (.so, .dylib, or .dll files) and invoke their functions as a processing step. This enables integration with external C/C++ libraries without requiring custom Go code or rebuilding Redpanda Connect. The processor uses `dlopen` (or platform equivalent) to load the specified library at runtime, looks up the function by name, and invokes it with arguments provided through Bloblang mapping. Function signatures are defined using the `signature` field, specifying parameter and return types. Results are returned as an array, where the first element is the return value (if not void), followed by any output parameters in their declaration order. > 📝 **NOTE** > > This component is available only in builds of Redpanda Connect that enable cgo. The standard release binaries, the Docker images, and the standard `rpk connect` plugin don’t include it. To use it, do one of the following: > > - Download the `redpanda-connect-cgo__linux_amd64` archive from the [GitHub releases page](https://github.com/redpanda-data/connect/releases). This archive is available for Linux AMD64 only. > > - Build Redpanda Connect from source with `CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata ./cmd/redpanda-connect`. The build needs a C compiler. > > > The cgo archive and the source build both need the ZeroMQ library (`libzmq`), because they include the `zmq4` components. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` The bloblang expression that returns an array of arguments to pass into the foreign function. **Type**: `string` ```yaml # Examples: args_mapping: root = [42, now().ts_unix_nano(), content()] ``` ### [](#function_name)`function_name` The name of the function to load from the shared library. **Type**: `string` ```yaml # Examples: function_name: MyExternCFunction ``` ### [](#library_path)`library_path` The path to the shared library (.so, .dylib or .dll) file to load dynamically. **Type**: `string` ```yaml # Examples: library_path: libbar.6.so # --- library_path: libfoo.dylib ``` ### [](#signature)`signature` The signature of the function. **Type**: `object` ### [](#signature-parameters)`signature.parameters[]` The parameters of the function. **Type**: `array` ### [](#signature-parameters-out)`signature.parameters[].out` If the parameter provided is an 'out' parameter, meaning if the function mutates the value, and the resulting value should be returned. This is only valid for pointer types. **Type**: `bool` **Default**: `false` ### [](#signature-parameters-type)`signature.parameters[].type` The data type of the parameter. **Type**: `string` | Option | Summary | | --- | --- | | byte* | A pointer to a byte array is provided as an argument. Note this byte array cannot be referenced once the function returns. args_mapping must return a byte array or string type for this argument, and the parameter in C for this should be void*. | | int32 | A 32 bit signed integer is provided as an argument | | int64 | A 64 bit signed integer is provided as an argument | ### [](#signature-return)`signature.return` The configuration for the function’s result. **Type**: `object` ### [](#signature-return-type)`signature.return.type` The data type of function’s return value **Type**: `string` | Option | Summary | | --- | --- | | int32 | A 32 bit signed integer is returned | | int64 | A 64 bit signed integer is returned | | void | The function returns nothing | ## [](#examples)Examples ### [](#call-a-libc-function)Call a libc function This is an example of loading libc.so.6 and calling a function on Linux. ```yaml pipeline: processors: - ffi: library_path: libc.so.6 function_name: memcmp args_mapping: 'root = ["foo", "bar", 3]' signature: return: type: int32 parameters: - type: byte* - type: byte* - type: int64 ``` --- # Page 235: for_each **URL**: https://docs.redpanda.com/connect/components/processors/for_each.md --- # for_each --- title: for_each latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/for_each page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/for_each.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/for_each.adoc description: A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message. This is useful for forcing batch wide processors such as [`dedupe`](https://docs.redpanda.com/connect/components/processors/dedupe/) or interpolations such as the `value` field of the `metadata` processor to execute on individual message parts of a batch instead. Please note that most processors already process per message of a batch, and this processor is not needed in those cases. ## Common ```yml processor: label: "" for_each: [] ``` ## Advanced ```yml processor: label: "" for_each: [] ``` --- # Page 236: gcp_bigquery_select **URL**: https://docs.redpanda.com/connect/components/processors/gcp_bigquery_select.md --- # gcp_bigquery_select --- title: gcp_bigquery_select latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/gcp_bigquery_select page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/gcp_bigquery_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/gcp_bigquery_select.adoc description: Executes a SELECT query against BigQuery and replaces messages with the rows returned. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a `SELECT` query against BigQuery and replaces messages with the rows returned. Introduced in version 3.64.0. #### Common ```yml processor: label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) job_labels: {} args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) job_labels: {} args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` ## [](#examples)Examples ### [](#word-count)Word count Given a stream of English terms, enrich the messages with the word count from Shakespeare’s public works: ```yaml pipeline: processors: - branch: processors: - gcp_bigquery_select: project: test-project table: bigquery-public-data.samples.shakespeare columns: - word - sum(word_count) as total_count where: word = ? suffix: | GROUP BY word ORDER BY total_count DESC LIMIT 10 args_mapping: root = [ this.term ] result_map: | root.count = this.get("0.total_count") ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that evaluates to an array of values for the `?` placeholders in the `where` field, in order. Each value is sent to BigQuery as a positional query parameter. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ### [](#credentials_json)`credentials_json` The Google Service Account credentials in JSON format (optional). Provide the contents of the credentials file as plain JSON, not Base64-encoded. Use this field to authenticate with Google Cloud services. If this field is empty, the component uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.33.0 or later. **Type**: `string` **Default**: `""` ### [](#job_labels)`job_labels` A map of labels to add to the query job. **Type**: `object` **Default**: `{}` ### [](#prefix)`prefix` Optional GoogleSQL text to add before the `SELECT` keyword of the generated query, for example a `WITH` clause. **Type**: `string` ### [](#project)`project` The GCP project where the query job runs. **Type**: `string` ### [](#suffix)`suffix` Optional GoogleSQL text to append after the generated query, for example `GROUP BY`, `ORDER BY`, or `LIMIT` clauses. **Type**: `string` ### [](#table)`table` The fully-qualified name of the BigQuery table to query. **Type**: `string` ```yaml # Examples: table: bigquery-public-data.samples.shakespeare ``` ### [](#where)`where` An optional `WHERE` clause to add to the query. The `args_mapping` field populates the placeholder arguments. Placeholders must always be question marks (`?`). **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 237: gcp_vertex_ai_chat **URL**: https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_chat.md --- # gcp_vertex_ai_chat --- title: gcp_vertex_ai_chat latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/gcp_vertex_ai_chat page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/gcp_vertex_ai_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/gcp_vertex_ai_chat.adoc description: Generates responses to messages in a chat conversation, using the Vertex AI API. page-git-created-date: "2024-09-04" page-git-modified-date: "2026-10-10" --- Generates responses to messages in a chat conversation, using the Vertex AI API. Introduced in version 4.34.0. This processor sends prompts to your chosen large language model (LLM) and generates text from the responses, using the Vertex AI API. For more information, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs). #### Common ```yml processor: label: "" gcp_vertex_ai_chat: project: "" # No default (required) credentials_json: "" # No default (optional) location: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) history: "" # No default (optional) attachment: "" # No default (optional) temperature: 0 # No default (optional) max_tokens: 0 # No default (optional) response_format: text tools: [] ``` #### Advanced ```yml processor: label: "" gcp_vertex_ai_chat: project: "" # No default (required) credentials_json: "" # No default (optional) location: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) attachment: "" # No default (optional) temperature: 0 # No default (optional) max_tokens: 0 # No default (optional) response_format: text top_p: 0 # No default (optional) top_k: 0 # No default (optional) stop: [] # No default (optional) presence_penalty: 0 # No default (optional) frequency_penalty: 0 # No default (optional) max_tool_calls: 10 tools: [] ``` ## [](#fields)Fields ### [](#attachment)`attachment` Additional data like an image to send with the prompt to the model. The result of the mapping must be a byte array, and the content type is automatically detected. Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: attachment: 'root = this.image.decode("base64") # decode base64 encoded image' ``` ### [](#credentials_json)`credentials_json` An optional field to set a Google Service Account Credentials JSON. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Historical messages to include in the chat request. The result of the bloblang query should be an array of objects of the form of \[{"role": "", "content":""}\], where role is "user" or "model". Requires version 4.56.0 or later. **Type**: `string` ### [](#location)`location` Specify the location of a fine tuned model. For base models, you can omit this field. **Type**: `string` ```yaml # Examples: location: us-central1 ``` ### [](#max_tokens)`max_tokens` The maximum number of output tokens to generate per message. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of sequential tool calls. Requires version 4.56.0 or later. **Type**: `int` **Default**: `10` ### [](#model)`model` The name of the LLM to use. For a full list of models, see the [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden). **Type**: `string` ```yaml # Examples: model: gemini-1.5-pro-001 # --- model: gemini-1.5-flash-001 ``` ### [](#presence_penalty)`presence_penalty` Positive values penalize new tokens if they appear in the text already, increasing the model’s likelihood to include new topics. **Type**: `float` ### [](#project)`project` The GCP project ID to use. **Type**: `string` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` The format of the generated response. You must also prompt the model to output the appropriate response type. **Type**: `string` **Default**: `text` **Options**: `text`, `json` ### [](#stop)`stop[]` Sets the stop sequences to use. When the model encounters one of these sequences, it stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Controls the randomness of predictions. **Type**: `float` ### [](#tools)`tools[]` The tools to allow the LLM to invoke. This allows building subpipelines that the LLM can choose to invoke to execute agentic-like actions. Requires version 4.56.0 or later. **Type**: `array` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool. The LLM uses it to decide whether to invoke the tool. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The parameters the LLM can provide when it invokes this tool, keyed by parameter name. **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` The values this parameter is limited to. Leave empty to accept any value. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The names of the parameters the LLM must provide when it invokes this tool. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The processors to run when the LLM invokes this tool. They receive a message whose payload is the tool call arguments as a JSON object, and their output is returned to the LLM as the tool result. **Type**: `array` ### [](#top_k)`top_k` Enables top-k sampling (optional). **Type**: `float` ### [](#top_p)`top_p` Enables nucleus sampling (optional). **Type**: `float` ## [](#examples)Examples ### [](#use-processors-as-tool-calls)Use processors as tool calls This example allows gemini to execute a subpipeline as a tool call to get more data. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - gcp_vertex_ai_chat: model: gemini-2.5-flash-preview-05-20 project: my-project location: us-central1 prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to lookup the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: # Spoof curl user-agent to get a plaintext text User-Agent: curl/8.11.1 output: stdout: {} ``` --- # Page 238: gcp_vertex_ai_embeddings **URL**: https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_embeddings.md --- # gcp_vertex_ai_embeddings --- title: gcp_vertex_ai_embeddings latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/gcp_vertex_ai_embeddings page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/gcp_vertex_ai_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/gcp_vertex_ai_embeddings.adoc description: Generates vector embeddings to represent input text, using the Vertex AI API. page-git-created-date: "2024-10-16" page-git-modified-date: "2026-10-10" --- Generates vector embeddings to represent input text, using the Vertex AI API. Introduced in version 4.37.0. This processor sends text strings to the Vertex AI API, which generates vector embeddings. By default, the processor submits the entire payload of each message as a string, unless you use the `text` configuration field to customize it. For more information, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings). #### Common ```yml processor: label: "" gcp_vertex_ai_embeddings: project: "" # No default (required) credentials_json: "" # No default (optional) location: us-central1 model: "" # No default (required) task_type: RETRIEVAL_DOCUMENT text: "" # No default (optional) output_dimensions: 0 # No default (optional) ``` #### Advanced ```yml processor: label: "" gcp_vertex_ai_embeddings: project: "" # No default (required) credentials_json: "" # No default (optional) location: us-central1 model: "" # No default (required) task_type: RETRIEVAL_DOCUMENT text: "" # No default (optional) output_dimensions: 0 # No default (optional) ``` ## [](#fields)Fields ### [](#credentials_json)`credentials_json` Set your Google Service Account Credentials as JSON (optional). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#location)`location` The location of the Vertex AI model that you want to use. **Type**: `string` **Default**: `us-central1` ### [](#model)`model` The name of the embedding model to use. For a full list of models, see the [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden). **Type**: `string` ```yaml # Examples: model: text-embedding-004 # --- model: text-multilingual-embedding-002 ``` ### [](#output_dimensions)`output_dimensions` The maximum length of a generated vector embedding. If this value is set, generated embeddings are truncated to this size. **Type**: `int` ### [](#project)`project` The ID of your Google Cloud project. **Type**: `string` ### [](#task_type)`task_type` Use the following options to optimize embeddings that the model generates for specific use cases. **Type**: `string` **Default**: `RETRIEVAL_DOCUMENT` | Option | Summary | | --- | --- | | CLASSIFICATION | optimize for being able classify texts according to preset labels | | CLUSTERING | optimize for clustering texts based on their similarities | | FACT_VERIFICATION | optimize for queries that are proving or disproving a fact such as "apples grow underground" | | QUESTION_ANSWERING | optimize for search proper questions such as "Why is the sky blue?" | | RETRIEVAL_DOCUMENT | optimize for documents that will be searched (also known as a corpus) | | RETRIEVAL_QUERY | optimize for queries such as "What is the best fish recipe?" or "best restaurant in Chicago" | | SEMANTIC_SIMILARITY | optimize for text similarity | ### [](#text)`text` The text you want to generate vector embeddings for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 239: google_drive_download **URL**: https://docs.redpanda.com/connect/components/processors/google_drive_download.md --- # google_drive_download --- title: google_drive_download latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/google_drive_download page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/google_drive_download.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/google_drive_download.adoc description: Downloads files from Google Drive. page-git-created-date: "2025-05-19" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Downloads files from Google Drive that contain matching file IDs. Try out the [example pipeline on this page](#examples), which downloads all files from your Google Drive. Introduced in version 4.53.0. #### Common ```yml processor: label: "" google_drive_download: credentials_json: "" # No default (optional) file_id: "" # No default (required) mime_type: "" # No default (required) shared_drives: false ``` #### Advanced ```yml processor: label: "" google_drive_download: credentials_json: "" # No default (optional) file_id: "" # No default (required) mime_type: "" # No default (required) export_mime_types: application/vnd.google-apps.document: "text/markdown" application/vnd.google-apps.drawing: "image/png" application/vnd.google-apps.presentation: "application/pdf" application/vnd.google-apps.script: "application/vnd.google-apps.script+json" application/vnd.google-apps.spreadsheet: "text/csv" shared_drives: false ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#export_mime_types)`export_mime_types` Maps Google Drive MIME types to [supported file export formats](https://developers.google.com/workspace/drive/api/guides/ref-export-formats). The MIME type is the key, and the export format is the value. **Type**: `object` **Default**: ```yaml application/vnd.google-apps.document: "text/markdown" application/vnd.google-apps.drawing: "image/png" application/vnd.google-apps.presentation: "application/pdf" application/vnd.google-apps.script: "application/vnd.google-apps.script+json" application/vnd.google-apps.spreadsheet: "text/csv" ``` ```yaml # Examples: export_mime_types: application/vnd.google-apps.document: application/pdf application/vnd.google-apps.drawing: application/pdf application/vnd.google-apps.presentation: application/pdf application/vnd.google-apps.spreadsheet: application/pdf # --- export_mime_types: application/vnd.google-apps.document: application/vnd.openxmlformats-officedocument.wordprocessingml.document application/vnd.google-apps.drawing: image/svg+xml application/vnd.google-apps.presentation: application/vnd.openxmlformats-officedocument.presentationml.presentation application/vnd.google-apps.spreadsheet: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet ``` ### [](#file_id)`file_id` The ID of the file to download from Google Drive. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#mime_type)`mime_type` The [MIME type](https://developers.google.com/workspace/drive/api/guides/mime-types) of the file for download. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#shared_drives)`shared_drives` Whether or not to include shared drives. Requires version 4.80.1 or later. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#download-files-from-google-drive)Download files from Google Drive This examples downloads all the files from Google Drive ```yaml pipeline: processors: - google_drive_search: query: "name = 'Test Doc'" - google_drive_download: file_id: "${!this.id}" mime_type: "${!this.mimeType}" ``` --- # Page 240: google_drive_list_labels **URL**: https://docs.redpanda.com/connect/components/processors/google_drive_list_labels.md --- # google_drive_list_labels --- title: google_drive_list_labels latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/google_drive_list_labels page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/google_drive_list_labels.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/google_drive_list_labels.adoc description: Lists labels for a file in Google Drive. page-git-created-date: "2025-05-19" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Lists [labels](https://developers.google.com/workspace/drive/api/guides/about-labels) for files on a Google Drive. Introduced in version 4.53.0. #### Common ```yml processor: label: "" google_drive_list_labels: credentials_json: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" google_drive_list_labels: credentials_json: "" # No default (optional) ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` --- # Page 241: google_drive_search **URL**: https://docs.redpanda.com/connect/components/processors/google_drive_search.md --- # google_drive_search --- title: google_drive_search latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/google_drive_search page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/google_drive_search.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/google_drive_search.adoc description: Searches Google Drive for files matching the provided query. page-git-created-date: "2025-05-19" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Searches Google Drive for files that match a specified query and emits the results as a batch of messages. Each message contains the [metadata of a Google Drive file](https://developers.google.com/workspace/drive/api/reference/rest/v3/files#File). Try out the [example pipeline on this page](#examples), which searches for and downloads all Google Drive files that match the specified query. Introduced in version 4.53.0. #### Common ```yml processor: label: "" google_drive_search: credentials_json: "" # No default (optional) query: "" # No default (required) projection: - "id" - "name" - "mimeType" - "size" - "labelInfo" include_label_ids: "" max_results: 64 shared_drives: false ``` #### Advanced ```yml processor: label: "" google_drive_search: credentials_json: "" # No default (optional) query: "" # No default (required) projection: - "id" - "name" - "mimeType" - "size" - "labelInfo" include_label_ids: "" max_results: 64 shared_drives: false ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#include_label_ids)`include_label_ids` A comma delimited list of label IDs to include in the Google Drive search result. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_results)`max_results` The maximum number of search results to return. **Type**: `int` **Default**: `64` ### [](#projection)`projection[]` Partial fields to include in the Google Drive search result. **Type**: `array` **Default**: ```yaml - "id" - "name" - "mimeType" - "size" - "labelInfo" ``` ### [](#query)`query` Specify a search query to locate matching files in Google Drive. This field supports: - The same query syntax as the Google Drive UI - [Bloblang interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) for dynamic query generation **Type**: `string` ### [](#shared_drives)`shared_drives` Whether or not to include shared drives in the result. Requires version 4.80.1 or later. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#search-download-files-from-google-drive)Search & download files from Google Drive This examples downloads all the files from Google Drive that are returned in the query ```yaml input: stdin: {} pipeline: processors: - google_drive_search: query: "${!content().string()}" - mutation: 'meta path = this.name' - google_drive_download: file_id: "${!this.id}" mime_type: "${!this.mimeType}" output: file: path: "${!@path}" codec: all-bytes ``` --- # Page 242: grok **URL**: https://docs.redpanda.com/connect/components/processors/grok.md --- # grok --- title: grok latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/grok page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/grok.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/grok.adoc description: Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values. Type hints within patterns are respected, therefore with the pattern `%\{WORD:first},%{INT:second:int}` and a payload of `foo,1` the resulting payload would be `\{"first":"foo","second":1}`. ## [](#performance)Performance This processor currently uses the [Go RE2](https://golang.org/s/re2syntax) regular expression engine, which is guaranteed to run in time linear to the size of the input. However, this property often makes it less performant than PCRE based implementations of grok. For more information, see [https://swtch.com/~rsc/regexp/regexp1.html](https://swtch.com/~rsc/regexp/regexp1.html). ### Common ```yml processor: label: "" grok: expressions: [] # No default (required) pattern_definitions: {} pattern_paths: [] ``` ### Advanced ```yml processor: label: "" grok: expressions: [] # No default (required) pattern_definitions: {} pattern_paths: [] named_captures_only: true use_default_patterns: true remove_empty_values: true ``` ## [](#examples)Examples ### [](#vpc-flow-logs)VPC Flow Logs Grok can be used to parse unstructured logs such as VPC flow logs that look like this: ```text 2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK ``` Into structured objects that look like this: ```json {"accountid":"123456789010","action":"ACCEPT","bytes":4249,"dstaddr":"172.31.16.21","dstport":22,"end":1418530070,"interfaceid":"eni-1235b8ca123456789","logstatus":"OK","packets":20,"protocol":6,"srcaddr":"172.31.16.139","srcport":20641,"start":1418530010,"version":2} ``` With the following config: ```yaml pipeline: processors: - grok: expressions: - '%{VPCFLOWLOG}' pattern_definitions: VPCFLOWLOG: '%{NUMBER:version:int} %{NUMBER:accountid} %{NOTSPACE:interfaceid} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:logstatus}' ``` ## [](#fields)Fields ### [](#expressions)`expressions[]` One or more Grok expressions to attempt against incoming messages. The first expression to match at least one value will be used to form a result. **Type**: `array` ### [](#named_captures_only)`named_captures_only` Whether to only capture values from named patterns. **Type**: `bool` **Default**: `true` ### [](#pattern_definitions)`pattern_definitions` A map of pattern definitions that can be referenced within `patterns`. **Type**: `object` **Default**: `{}` ### [](#pattern_paths)`pattern_paths[]` A list of paths to load Grok patterns from. This field supports wildcards, including super globs (double star). **Type**: `array` **Default**: `[]` ### [](#remove_empty_values)`remove_empty_values` Whether to remove values that are empty from the resulting structure. **Type**: `bool` **Default**: `true` ### [](#use_default_patterns)`use_default_patterns` Whether to use a [default set of patterns](#default-patterns). **Type**: `bool` **Default**: `true` ## [](#default-patterns)Default patterns For summary of the default patterns on offer, see [https://github.com/Jeffail/grok/blob/master/patterns.go#L5](https://github.com/Jeffail/grok/blob/master/patterns.go#L5). --- # Page 243: group_by_value **URL**: https://docs.redpanda.com/connect/components/processors/group_by_value.md --- # group_by_value --- title: group_by_value latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/group_by_value page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/group_by_value.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/group_by_value.adoc description: Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a function interpolated string evaluated per message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a [function interpolated string](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) evaluated per message. This allows you to group messages using arbitrary fields within their content or metadata, process them individually, and send them to unique locations as per their group. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](https://docs.redpanda.com/connect/configuration/batching/). To further divide each group into individual messages, follow this processor with a [`split` processor](https://docs.redpanda.com/connect/components/processors/split/). ### Common ```yml processor: label: "" group_by_value: value: "" # No default (required) ``` ### Advanced ```yml processor: label: "" group_by_value: value: "" # No default (required) ``` ## [](#fields)Fields ### [](#value)`value` The interpolated string to group based on. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: value: ${! meta("kafka_key") } # --- value: ${! json("foo.bar") }-${! meta("baz") } ``` ## [](#examples)Examples If we were consuming Kafka messages and needed to group them by their key, archive the groups, and send them to S3 with the key as part of the path we could achieve that with the following: ```yaml pipeline: processors: - group_by_value: value: ${! meta("kafka_key") } - archive: format: tar - compress: algorithm: gzip output: aws_s3: bucket: TODO path: docs/${! meta("kafka_key") }/${! count("files") }-${! timestamp_unix_nano() }.tar.gz ``` --- # Page 244: group_by **URL**: https://docs.redpanda.com/connect/components/processors/group_by.md --- # group_by --- title: group_by latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/group_by page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/group_by.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/group_by.adoc description: Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a Bloblang query. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Splits a [batch of messages](https://docs.redpanda.com/connect/configuration/batching/) into N batches, where each resulting batch contains a group of messages determined by a [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/). Once the groups are established a list of processors are applied to their respective grouped batch, which can be used to label the batch as per their grouping. Messages that do not pass the check of any specified group are placed in their own group. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](https://docs.redpanda.com/connect/configuration/batching/). To further divide each group into individual messages, follow this processor with a [`split` processor](https://docs.redpanda.com/connect/components/processors/split/). #### Common ```yml processor: label: "" group_by: - check: "" # No default (required) processors: [] ``` #### Advanced ```yml processor: label: "" group_by: - check: "" # No default (required) processors: [] ``` ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message belongs to a given group. **Type**: `string` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") # --- check: "true" ``` ### [](#processors)`processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to execute on the newly formed group. **Type**: `array` **Default**: `[]` ## [](#examples)Examples ### [](#grouped-processing)Grouped Processing Imagine we have a batch of messages that we wish to split into a group of foos and everything else, which should be sent to different output destinations based on those groupings. We also need to send the foos as a tar gzip archive. For this purpose we can use the `group_by` processor with a [`switch`](https://docs.redpanda.com/connect/components/outputs/switch/) output: ```yaml pipeline: processors: - group_by: - check: content().contains("this is a foo") processors: - archive: format: tar - compress: algorithm: gzip - mapping: 'meta grouping = "foo"' output: switch: cases: - check: meta("grouping") == "foo" output: gcp_pubsub: project: foo_prod topic: only_the_foos - output: gcp_pubsub: project: somewhere_else topic: no_foos_here ``` --- # Page 245: http **URL**: https://docs.redpanda.com/connect/components/processors/http.md --- # http --- title: http latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/http page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/http.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/http.adoc description: Performs an HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs a HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response. #### Common ```yml processor: label: "" http: url: "" # No default (required) verb: POST headers: {} rate_limit: "" # No default (optional) timeout: 5s parallel: false ``` #### Advanced ```yml processor: label: "" http: url: "" # No default (required) verb: POST headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false batch_as_multipart: false parallel: false ``` ## [](#rate-limit-requests)Rate limit requests You can use the `rate_limit` field to specify a [rate limit resource](https://docs.redpanda.com/connect/components/rate_limits/about/), which restricts the number of requests processed service-wide, regardless of how many components you run in parallel. ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#map-payloads-with-the-branch-processor)Map payloads with the branch processor You can use the [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) to transform or encode the payload into a specific request body format, and map the response back into the original payload instead of replacing it entirely. This example uses a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) to strip the request message into an empty body (`request_map: 'root = ""'`), grab an HTTP payload, and place the result back into the original message at the path `repo.status`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: 'root.repo.status = this' ``` ## [](#response-codes)Response codes HTTP response codes in the 200-299 range indicate a successful response. You can use the [`successful_on`](#successful_on) field to add more success status codes. HTTP status codes in the 300-399 range are redirects. The [`follow_redirects` field](#follow_redirects) determines how these responses are handled. If a request returns a response code that matches an entry in: - The [`backoff_on` field](#backoff_on), the request is retried after increasing intervals. - The [`drop_on` field](#drop_on), the request is immediately treated as a failure. ## [](#add-metadata-to-errors)Add metadata to errors If a request returns an error response code, this processor sets a `http_status_code` metadata field in the resulting message. > 💡 **TIP** > > You can use the [`extract_headers`](#extract_headers) field to define rules for copying headers into messages generated from the response. ## [](#error-handling)Error handling When all retry attempts for a message are exhausted, this processor cancels the attempt. By default, the failed message continues through the pipeline unchanged unless you configure other error-handling. For example, you might want to drop failed messages or route them to a dead letter queue. For more information, see [Error Handling](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#fields)Fields ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure and trigger retries with an increasing backoff period between attempts. **Type**: `array` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#batch_as_multipart)`batch_as_multipart` When set to `true`, sends all messages in a batch as a single request using [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). When set to `false`, sends messages in a batch as individual requests. **Type**: `bool` **Default**: `false` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. Requires version 4.47.0 or later. **Type**: `bool` **Default**: `false` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure where retries should not be attempted. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `array` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, `""` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether to transparently follow redirects, that is, responses with HTTP status codes in the 300-399 range. If set to `false`, the response message includes the body, status, and headers from the redirect response, and the component does not make a request to the URL specified in the `Location` header. Requires version 4.39.0 or later. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys should be added to the HTTP request as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. Requires version 4.21.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). Requires version 3.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#parallel)`parallel` When processing batched messages, this field determines whether messages in the batch are sent in parallel. If set to `false`, messages are sent serially. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` An HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](https://docs.redpanda.com/connect/components/rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The period to wait between failed requests before retrying. For status codes listed in `backoff_on`, this is the starting period, which increases with each attempt up to `max_retry_backoff`. **Type**: `string` **Default**: `1s` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in the `backoff_on` or `drop_on` fields, regardless of this field. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `POST` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` ## [](#examples)Examples ### [](#branched-request)Branched Request This example uses a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) to strip the request message into an empty body, grab an HTTP payload, and place the result back into the original message at the path `repo.status`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: 'root.repo.status = this' ``` --- # Page 246: insert_part **URL**: https://docs.redpanda.com/connect/components/processors/insert_part.md --- # insert_part --- title: insert_part latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/insert_part page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/insert_part.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/insert_part.adoc description: Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end. The index can be negative, and if so the message will be inserted from the end counting backwards starting from -1. E.g. if index = -1 then the new message will become the last of the batch, if index = -2 then the new message will be inserted before the last message, and so on. If the negative index is greater than the length of the existing batch it will be inserted at the beginning. The new message will have metadata copied from the first pre-existing message of the batch. This processor will interpolate functions within the 'content' field, you can find a list of functions [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). #### Common ```yml processor: label: "" insert_part: index: -1 content: "" ``` #### Advanced ```yml processor: label: "" insert_part: index: -1 content: "" ``` ## [](#fields)Fields ### [](#content)`content` The content of the message being inserted. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#index)`index` The index within the batch to insert the message at. **Type**: `int` **Default**: `-1` --- # Page 247: javascript **URL**: https://docs.redpanda.com/connect/components/processors/javascript.md --- # javascript --- title: javascript latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/javascript page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/javascript.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/javascript.adoc description: Executes a provided JavaScript code block or file for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a JavaScript code block or file for each message. Introduced in version 4.14.0. ##### Common ```yml processor: label: "" javascript: code: "" # No default (optional) file: "" # No default (optional) global_folders: [] ``` ##### Advanced ```yml processor: label: "" javascript: code: "" # No default (optional) file: "" # No default (optional) global_folders: [] ``` The [execution engine](https://github.com/dop251/goja) behind this processor provides full ECMAScript 5.1 support (including regex and strict mode). Most of the ECMAScript 6 spec is implemented but this is a work in progress. Imports via `require` should work similarly to NodeJS, and access to the console is supported which will print via the Redpanda Connect logger. More caveats can be found on [GitHub](https://github.com/dop251/goja#known-incompatibilities-and-caveats). This processor is implemented using the [github.com/dop251/goja](https://github.com/dop251/goja) library. ## [](#fields)Fields ### [](#code)`code` An inline JavaScript program to run. You must specify a value for either the `code` or `file` field. **Type**: `string` ### [](#file)`file` A file containing a JavaScript program to run. You must specify a value for either the `code` or `file` field. **Type**: `string` ### [](#global_folders)`global_folders[]` A list of directories to load modules from if the requested JavaScript module is not found elsewhere. **Type**: `array` **Default**: `[]` ## [](#examples)Examples ### [](#simple-mutation)Simple mutation In this example we define a simple function that performs a basic mutation against messages, treating their contents as raw strings. ```yaml pipeline: processors: - javascript: code: 'benthos.v0_msg_set_string(benthos.v0_msg_as_string() + "hello world");' ``` ### [](#structured-mutation)Structured mutation In this example we define a function that performs basic mutations against a structured message. Note that we encapsulate the logic within an anonymous function that is called for each invocation, this is required in order to avoid duplicate variable declarations in the global state. ```yaml pipeline: processors: - javascript: code: | (() => { let thing = benthos.v0_msg_as_structured(); thing.num_keys = Object.keys(thing).length; delete thing["b"]; benthos.v0_msg_set_structured(thing); })(); ``` ## [](#runtime)Runtime To optimize code execution, JavaScript runtimes are created on demand (in order to support parallel execution) and are reused across invocations. Therefore, it’s important to understand that the global state created by your programs will outlive individual invocations. For your programs to avoid failing after the first invocation, ensure that you do not define variables at the global scope. Although technically possible, it is recommended that you do not rely on the global state for maintaining state across invocations as the pooling nature of the runtimes will prevent deterministic behavior. We aim to support deterministic strategies for mutating global state in the future. ## [](#functions)Functions ### [](#benthos-v0_fetch)`benthos.v0_fetch` Executes an HTTP request synchronously and returns the result as an object of the form `{"status":200,"body":"foo"}`. #### [](#parameters)Parameters **`url`** The URL to fetch **`headers`** An object of string/string key/value pairs to add the request as headers. **`method`** The method of the request. **`body`** <(optional) string> A body to send. #### [](#examples-2)Examples ```javascript let result = benthos.v0_fetch("http://example.com", {}, "GET", "") benthos.v0_msg_set_structured(result); ``` ### [](#benthos-v0_msg_as_string)`benthos.v0_msg_as_string` Obtain the raw contents of the processed message as a string. #### [](#examples-3)Examples ```javascript let contents = benthos.v0_msg_as_string(); ``` ### [](#benthos-v0_msg_as_structured)`benthos.v0_msg_as_structured` Obtain the root of the processed message as a structured value. If the message is not valid JSON or has not already been expanded into a structured form this function will throw an error. #### [](#examples-4)Examples ```javascript let foo = benthos.v0_msg_as_structured().foo; ``` ### [](#benthos-v0_msg_exists_meta)`benthos.v0_msg_exists_meta` Check that a metadata key exists. #### [](#parameters-2)Parameters **`name`** The metadata key to search for. #### [](#examples-5)Examples ```javascript if (benthos.v0_msg_exists_meta("kafka_key")) {} ``` ### [](#benthos-v0_msg_get_meta)`benthos.v0_msg_get_meta` Get the value of a metadata key from the processed message. #### [](#parameters-3)Parameters **`name`** The metadata key to search for. #### [](#examples-6)Examples ```javascript let key = benthos.v0_msg_get_meta("kafka_key"); ``` ### [](#benthos-v0_msg_set_meta)`benthos.v0_msg_set_meta` Set a metadata key on the processed message to a value. #### [](#parameters-4)Parameters **`name`** The metadata key to set. **`value`** The value to set it to. #### [](#examples-7)Examples ```javascript benthos.v0_msg_set_meta("thing", "hello world"); ``` ### [](#benthos-v0_msg_set_string)`benthos.v0_msg_set_string` Set the contents of the processed message to a given string. #### [](#parameters-5)Parameters **`value`** The value to set it to. #### [](#examples-8)Examples ```javascript benthos.v0_msg_set_string("hello world"); ``` ### [](#benthos-v0_msg_set_structured)`benthos.v0_msg_set_structured` Set the root of the processed message to a given value of any type. #### [](#parameters-6)Parameters **`value`** The value to set it to. #### [](#examples-9)Examples ```javascript benthos.v0_msg_set_structured({ "foo": "a thing", "bar": "something else", "baz": 1234 }); ``` --- # Page 248: jira **URL**: https://docs.redpanda.com/connect/components/processors/jira.md --- # jira --- title: jira latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/jira page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/jira.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/jira.adoc description: Queries Jira resources and returns structured data. page-git-created-date: "2025-11-03" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > ⚠️ **WARNING: Deprecated in 4.100.0** > > Deprecated in 4.100.0 > > This component is deprecated and will be removed in the next major version release. To stream Jira issues, comments, or changelog entries, use the [`jira` input](https://docs.redpanda.com/connect/components/inputs/jira/) instead. Queries Jira resources and returns structured data. Introduced in version 4.68.0. ### Common ```yaml processor: label: "" jira: username: "" # No default (required) api_token: "" # No default (required) max_results_per_page: 50 base_url: "" # No default (required) timeout: 5s ``` ### Advanced ```yaml processor: label: "" jira: username: "" # No default (required) api_token: "" # No default (required) max_results_per_page: 50 base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically. Supports querying the following Jira resources: - Issues (JQL queries) - Issue transitions - Users - Roles - Project versions - Project categories - Project types - Projects The processor authenticates using basic authentication with username and API token. Input messages should contain valid Jira queries in JSON format. Set `base_url` to the URL of your Jira instance, for example `[https://your-domain.atlassian.net](https://your-domain.atlassian.net)`. ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. Requires version 4.83.0 or later. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. Requires version 4.83.0 or later. **Type**: `string` **Default**: `""` **Options**: `""`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#api_token)`api_token` The Jira API token for the specified account. You can generate an API token from your [Atlassian account settings](https://id.atlassian.com/manage-profile/security/api-tokens). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. Requires version 4.83.0 or later. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` The base URL of the target service (for example, `[https://api.example.com](https://api.example.com)`). TLS is enabled automatically for `https` URLs. **Type**: `string` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. Requires version 4.83.0 or later. **Type**: `bool` **Default**: `false` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. Requires version 4.83.0 or later. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#max_results_per_page)`max_results_per_page` The maximum number of results to return per page when calling the Jira API. [Pagination](https://docs.atlassian.com/software/jira/docs/api/REST/9.17.0/#pagination) in the Jira API is zero-based, so the first page starts at `0`. **Type**: `int` **Default**: `50` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. Requires version 4.83.0 or later. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.83.0 or later. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. Requires version 4.83.0 or later. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. Requires version 4.83.0 or later. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. Requires version 4.83.0 or later. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. Requires version 4.83.0 or later. **Type**: `float` **Default**: `0` ### [](#username)`username` The username or email address of the Jira account. **Type**: `string` ## [](#examples)Examples ### [](#minimal-configuration)Minimal configuration Basic Jira processor setup with required fields only ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" ``` ### [](#full-configuration-with-tuning)Full configuration with tuning Complete configuration with pagination and timeout settings ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 200 timeout: "30s" ``` ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 249: jmespath **URL**: https://docs.redpanda.com/connect/components/processors/jmespath.md --- # jmespath --- title: jmespath latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/jmespath page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/jmespath.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/jmespath.adoc description: Executes a JMESPath query on JSON documents and replaces the message with the resulting document. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a [JMESPath query](http://jmespath.org/) on JSON documents and replaces the message with the resulting document. > 💡 **TIP: Try out Bloblang** > > Try out Bloblang > > For better performance and improved capabilities try native Redpanda Connect mapping with the [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/). #### Common ```yml processor: label: "" jmespath: query: "" # No default (required) ``` #### Advanced ```yml processor: label: "" jmespath: query: "" # No default (required) ``` ## [](#fields)Fields ### [](#query)`query` The JMESPath query to apply to messages. **Type**: `string` ## [](#examples)Examples ### [](#mapping)Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - jmespath: query: "locations[?state == 'WA'].name | sort(@) | {Cities: join(', ', @)}" ``` --- # Page 250: jq **URL**: https://docs.redpanda.com/connect/components/processors/jq.md --- # jq --- title: jq latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/jq page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/jq.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/jq.adoc description: Transforms and filters messages using jq queries. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Transforms and filters messages using jq queries. #### Common ```yml processor: label: "" jq: query: "" # No default (required) ``` #### Advanced ```yml processor: label: "" jq: query: "" # No default (required) raw: false output_raw: false ``` > 💡 **TIP: Try out Bloblang** > > Try out Bloblang > > For better performance and improved capabilities try out native Redpanda Connect mapping with the [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/). The provided query is executed on each message, targeting either the contents as a structured JSON value or as a raw string using the field `raw`, and the message is replaced with the query result. Message metadata is also accessible within the query from the variable `$metadata`. This processor uses the [gojq library](https://github.com/itchyny/gojq), and therefore does not require jq to be installed as a dependency. However, this also means there are some [differences in how these queries are executed](https://github.com/itchyny/gojq#difference-to-jq) versus the jq cli. If the query does not emit any value then the message is filtered, if the query returns multiple values then the resulting message will be an array containing all values. The full query syntax is described in [jq’s documentation](https://stedolan.github.io/jq/manual/). ## [](#error-handling)Error handling Queries can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#fields)Fields ### [](#output_raw)`output_raw` Whether to output raw text (unquoted) instead of JSON strings when the emitted values are string types. **Type**: `bool` **Default**: `false` ### [](#query)`query` The jq query to filter and transform messages with. **Type**: `string` ### [](#raw)`raw` Whether to process the input as a raw string instead of as JSON. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#mapping)Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - jq: query: '{Cities: .locations | map(select(.state == "WA").name) | sort | join(", ") }' ``` --- # Page 251: json_schema **URL**: https://docs.redpanda.com/connect/components/processors/json_schema.md --- # json_schema --- title: json_schema latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/json_schema page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/json_schema.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/json_schema.adoc description: Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances. If a message does not match the schema it can be caught using error handling methods. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances. If a message does not match the schema it can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). Please refer to the [JSON Schema website](https://json-schema.org/) for information and tutorials regarding the syntax of the schema. ### Common ```yml processor: label: "" json_schema: schema: "" # No default (optional) schema_path: "" # No default (optional) ``` ### Advanced ```yml processor: label: "" json_schema: schema: "" # No default (optional) schema_path: "" # No default (optional) ``` ## [](#fields)Fields ### [](#schema)`schema` A schema to apply. Use either this or the `schema_path` field. **Type**: `string` ### [](#schema_path)`schema_path` The path of a schema document to apply. Use either this or the `schema` field. **Type**: `string` ## [](#examples)Examples With the following JSONSchema document: ```json { "$id": "https://example.com/person.schema.json", "$schema": "http://json-schema.org/draft-07/schema#", "title": "Person", "type": "object", "properties": { "firstName": { "type": "string", "description": "The person's first name." }, "lastName": { "type": "string", "description": "The person's last name." }, "age": { "description": "Age in years which must be equal to or greater than zero.", "type": "integer", "minimum": 0 } } } ``` And the following Redpanda Connect configuration: ```yaml pipeline: processors: - json_schema: schema_path: "file://path_to_schema.json" - catch: - log: level: ERROR message: "Schema validation failed due to: ${!error()}" - mapping: 'root = deleted()' # Drop messages that fail ``` If a payload being processed looked like: ```json {"firstName":"John","lastName":"Doe","age":-21} ``` Then a log message would appear explaining the fault and the payload would be dropped. --- # Page 252: log **URL**: https://docs.redpanda.com/connect/components/processors/log.md --- # log --- title: log latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/log page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/log.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/log.adoc description: Prints a log event for each message. Messages always remain unchanged. The log message can be set using function interpolations described in Bloblang queries which allows you to log the contents and metadata of messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Prints a log event for each message. Messages always remain unchanged. The log message can be set using function interpolations described in [Bloblang queries](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) which allows you to log the contents and metadata of messages. The `level` field determines the log level of the printed events and can be any of the following values: TRACE, DEBUG, INFO, WARN, ERROR. ## [](#structured-fields)Structured fields It’s also possible add custom fields to logs when the format is set to a structured form such as `json` or `logfmt` with the config field [`fields_mapping`](#fields_mapping): ```yaml pipeline: processors: - log: level: DEBUG message: hello world fields_mapping: | root.reason = "cus I wana" root.id = this.id root.age = this.user.age root.kafka_topic = meta("kafka_topic") ``` ### Common ```yml processor: label: "" log: level: INFO fields_mapping: "" # No default (optional) message: "" ``` ### Advanced ```yml processor: label: "" log: level: INFO fields_mapping: "" # No default (optional) message: "" ``` ## [](#fields)Fields ### [](#fields_mapping)`fields_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that can be used to specify extra fields to add to the log. If log fields are also added with the `fields` field then those values will override matching keys from this mapping. **Type**: `string` ```yaml # Examples: fields_mapping: |- root.reason = "cus I wana" root.id = this.id root.age = this.user.age.number() root.kafka_topic = meta("kafka_topic") ``` ### [](#level)`level` The log level to use. **Type**: `string` **Default**: `INFO` **Options**: `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE` ### [](#message)`message` The message to print. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 253: mapping **URL**: https://docs.redpanda.com/connect/components/processors/mapping.md --- # mapping --- title: mapping latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/mapping page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/mapping.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/mapping.adoc description: Executes a Bloblang mapping on messages, creating a new document that replaces (or filters) the original message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) mapping on messages, creating a new document that replaces (or filters) the original message. Introduced in version 4.5.0. #### Common ```yml processor: label: "" mapping: "" # No default (required) ``` #### Advanced ```yml processor: label: "" mapping: "" # No default (required) ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information, see [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. Note: This processor is equivalent to the [Bloblang](https://docs.redpanda.com/connect/components/processors/bloblang/#component-rename) one. The latter will be deprecated in a future release. ## [](#input-document-immutability)Input document immutability Mapping operates by creating an entirely new object during assignments, this has the advantage of treating the original referenced document as immutable and therefore queryable at any stage of your mapping. For example, with the following mapping: ```bloblang root.id = this.id root.invitees = this.invitees.filter(i -> i.mood >= 0.5) root.rejected = this.invitees.filter(i -> i.mood < 0.5) # In: {"id":"party-2024","invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Notice that we mutate the value of `invitees` in the resulting document by filtering out objects with a lower mood. However, even after doing so we’re still able to reference the unchanged original contents of this value from the input document in order to populate a second field. Within this mapping we also have the flexibility to reference the mutable mapped document by using the keyword `root` (i.e. `root.invitees`) on the right-hand side instead. Mapping documents is advantageous in situations where the result is a document with a dramatically different shape to the input document, since we are effectively rebuilding the document in its entirety and might as well keep a reference to the unchanged input document throughout. However, in situations where we are only performing minor alterations to the input document, the rest of which is unchanged, it might be more efficient to use the [`mutation` processor](https://docs.redpanda.com/connect/components/processors/mutation/) instead. ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired [fallback behavior](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the documents down to just the ID and only those fans with an obsession score above 0.5, giving us: ```json { "id":"foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - mapping: | root.id = this.id root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - mapping: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` --- # Page 254: metric **URL**: https://docs.redpanda.com/connect/components/processors/metric.md --- # metric --- title: metric latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/metric page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/metric.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/metric.adoc description: Emit custom metrics by extracting values from messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Emit custom metrics by extracting values from messages. This processor works by evaluating an [interpolated field `value`](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) for each message and updating a emitted metric according to the [type](#types). Custom metrics such as these are emitted along with Redpanda Connect internal metrics, where you can customize where metrics are sent, which metric names are emitted and rename them as/when appropriate. For more information see the [metrics docs](https://docs.redpanda.com/connect/components/metrics/about/). #### Common ```yml processor: label: "" metric: type: "" # No default (required) name: "" # No default (required) labels: {} # No default (optional) value: "" ``` #### Advanced ```yml processor: label: "" metric: type: "" # No default (required) name: "" # No default (required) labels: {} # No default (optional) value: "" ``` ## [](#fields)Fields ### [](#labels)`labels` A map of label names and values that can be used to enrich metrics. Labels are not supported by some metric destinations, in which case the metrics series are combined. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` ```yaml # Examples: labels: topic: ${! meta("kafka_topic") } type: ${! json("doc.type") } ``` ### [](#name)`name` The name of the metric to create, this must be unique across all Redpanda Connect components otherwise it will overwrite those other metrics. **Type**: `string` ### [](#type)`type` The metric [type](#types) to create. **Type**: `string` **Options**: `counter`, `counter_by`, `gauge`, `timing` ### [](#value)`value` For some metric types specifies a value to set, increment. Certain metrics exporters such as Prometheus support floating point values, but those that do not will cast a floating point value into an integer. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#counter)Counter In this example we emit a counter metric called `Foos`, which increments for every message processed, and we label the metric with some metadata about where the message came from and a field from the document that states what type it is. We also configure our metrics to emit to CloudWatch, and explicitly only allow our custom metric and some internal Redpanda Connect metrics to emit. ```yaml pipeline: processors: - metric: name: Foos type: counter labels: topic: ${! meta("kafka_topic") } partition: ${! meta("kafka_partition") } type: ${! json("document.type").or("unknown") } metrics: mapping: | root = if ![ "Foos", "input_received", "output_sent" ].contains(this) { deleted() } aws_cloudwatch: namespace: ProdConsumer ``` ### [](#gauge)Gauge In this example we emit a gauge metric called `FooSize`, which is given a value extracted from JSON messages at the path `foo.size`. We then also configure our Prometheus metric exporter to only emit this custom metric and nothing else. We also label the metric with some metadata. ```yaml pipeline: processors: - metric: name: FooSize type: gauge labels: topic: ${! meta("kafka_topic") } value: ${! json("foo.size") } metrics: mapping: 'if this != "FooSize" { deleted() }' prometheus: {} ``` ## [](#types)Types ### [](#counter-2)`counter` Increments a counter by exactly 1, the contents of `value` are ignored by this type. ### [](#counter_by)`counter_by` If the contents of `value` can be parsed as a positive integer value then the counter is incremented by this value. For example, the following configuration will increment the value of the `count.custom.field` metric by the contents of `field.some.value`: ```yaml pipeline: processors: - metric: type: counter_by name: CountCustomField value: ${!json("field.some.value")} ``` ### [](#gauge-2)`gauge` If the contents of `value` can be parsed as a positive integer value then the gauge is set to this value. For example, the following configuration will set the value of the `gauge.custom.field` metric to the contents of `field.some.value`: ```yaml pipeline: processors: - metric: type: gauge name: GaugeCustomField value: ${!json("field.some.value")} ``` ### [](#timing)`timing` Equivalent to `gauge` where instead the metric is a timing. It is recommended that timing values are recorded in nanoseconds in order to be consistent with standard Redpanda Connect timing metrics, as in some cases these values are automatically converted into other units such as when exporting timings as histograms with Prometheus metrics. --- # Page 255: mongodb **URL**: https://docs.redpanda.com/connect/components/processors/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/mongodb page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/mongodb.adoc description: Performs operations against MongoDB for each message, allowing you to store or retrieve data within message payloads. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs operations against MongoDB for each message, allowing you to store or retrieve data within message payloads. Introduced in version 3.43.0. #### Common ```yml processor: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) operation: insert-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false ``` #### Advanced ```yml processor: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos aws: enabled: false region: "" # No default (optional) session_duration: 1h id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) collection: "" # No default (required) operation: insert-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false json_marshal_mode: canonical ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. Requires version 4.32.0 or later. **Type**: `string` **Default**: `benthos` ### [](#aws)`aws` AWS IAM authentication using the `MONGODB-AWS` mechanism, for example against MongoDB Atlas. When enabled, IAM credentials are used instead of a static username and password. Role-derived session credentials are resolved when the component connects and are re-resolved whenever it reconnects. The `mongodb` processor and cache establish their client once at creation and cannot refresh expiring session credentials, so `role`, `roles` and session tokens are rejected for those components; use the ambient credential chain or long-lived access keys with them. For long-running pipelines, prefer the ambient credential chain (leave keys and roles unset), which the driver refreshes automatically. Requires version 4.106.0 or later. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication using the driver-native `MONGODB-AWS` mechanism. The MongoDB Atlas database user must be created with the AWS IAM authentication type, and connections require TLS. When no static credentials or roles are configured, the ambient AWS credential chain (environment variables, EC2 instance profile, EKS pod role) is used and expiring credentials are refreshed automatically. **Type**: `bool` **Default**: `false` ### [](#aws-id)`aws.id` The AWS access key ID to authenticate with. When empty, the default AWS credential chain is used. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region used when assuming roles (for STS calls). Only used when `role` or `roles` are configured; the ambient and static-key paths ignore it. If no region is specified then the environment default is used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Cannot be combined with `roles`; use the `roles` array instead when chaining multiple roles. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID to use when assuming the role set in `role`. Each entry in `roles` sets its own external ID. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles are assumed in sequence, each using the credentials of the previous one, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. Cannot be combined with `role`. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The AWS secret access key that pairs with `id`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#aws-session_duration)`aws.session_duration` The duration of the STS session requested when assuming roles. AWS requires at least 15 minutes and caps sessions created through role chaining at one hour. Only used when `role` or `roles` are configured. For long-running pipelines, prefer the ambient credential chain over a fixed session duration, since the driver refreshes ambient credentials automatically as they near expiry. **Type**: `string` **Default**: `1h` ### [](#aws-token)`aws.token` The AWS session token to use with `id` and `secret`. Required only when using short-term credentials. **Type**: `string` ### [](#collection)`collection` The name of the target collection. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#document_map)`document_map` A Bloblang map that represents a document to store in MongoDB, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, `update-one`, and `aggregate`. **Type**: `string` **Default**: `""` ```yaml # Examples: document_map: |- root.a = this.foo root.b = this.bar ``` ### [](#filter_map)`filter_map` A Bloblang map that represents a filter for a MongoDB command, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `filter_map` parameter is required for all database operations except `insert-one`. The `filter_map` is used to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion. **Type**: `string` **Default**: `""` ```yaml # Examples: filter_map: |- root.a = this.foo root.b = this.bar ``` ### [](#hint_map)`hint_map` A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). This map is optional, and is used with all operations except `insert-one`. Define a `hint_map` to improve performance when finding documents in the MongoDB database. **Type**: `string` **Default**: `""` ```yaml # Examples: hint_map: |- root.a = this.foo root.b = this.bar ``` ### [](#json_marshal_mode)`json_marshal_mode` Controls the format used to convert BSON documents to JSON in the output message. Requires version 3.60.0 or later. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation. That is, conversion from relaxed format to BSON can lose type information. | ### [](#operation)`operation` The MongoDB database operation to perform. **Type**: `string` **Default**: `insert-one` **Options**: `insert-one`, `delete-one`, `delete-many`, `replace-one`, `update-one`, `find-one`, `aggregate` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#upsert)`upsert` The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created. Requires version 3.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` ### [](#write_concern)`write_concern` The [write concern settings](https://www.mongodb.com/docs/manual/reference/write-concern/) for the MongoDB connection. **Type**: `object` ### [](#write_concern-j)`write_concern.j` The `j` option requests acknowledgement from MongoDB that write operations are written to the journal. **Type**: `bool` **Default**: `false` ### [](#write_concern-w)`write_concern.w` The `w` option requests acknowledgement that write operations propagate to the specified number of MongoDB instances. Set it to `majority` to wait for a calculated majority of nodes to acknowledge the write operation, to an integer to specify the minimum number of nodes that must acknowledge the operation, or to the name of a custom write concern configured in the cluster. **Type**: `string` **Default**: `majority` ### [](#write_concern-w_timeout)`write_concern.w_timeout` The write concern timeout. **Type**: `string` **Default**: `""` --- # Page 256: msgpack **URL**: https://docs.redpanda.com/connect/components/processors/msgpack.md --- # msgpack --- title: msgpack latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/msgpack page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/msgpack.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/msgpack.adoc description: Converts messages to or from the MessagePack format. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Converts messages to or from the [MessagePack](https://msgpack.org/) format. Introduced in version 3.59.0. #### Common ```yml processor: label: "" msgpack: operator: "" # No default (required) ``` #### Advanced ```yml processor: label: "" msgpack: operator: "" # No default (required) ``` ## [](#fields)Fields ### [](#operator)`operator` The operation to perform on messages. **Type**: `string` | Option | Summary | | --- | --- | | from_json | Convert JSON messages to MessagePack format | | to_json | Convert MessagePack messages to JSON format | --- # Page 257: mutation **URL**: https://docs.redpanda.com/connect/components/processors/mutation.md --- # mutation --- title: mutation latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/mutation page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/mutation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/mutation.adoc description: Executes a Bloblang mapping and directly transforms the contents of messages, mutating (or deleting) them. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) mapping and directly transforms the contents of messages, mutating (or deleting) them. Introduced in version 4.5.0. #### Common ```yml processor: label: "" mutation: "" # No default (required) ``` #### Advanced ```yml processor: label: "" mutation: "" # No default (required) ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information, see [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. ## [](#input-document-mutability)Input document mutability A mutation is a mapping that transforms input documents directly, this has the advantage of reducing the need to copy the data fed into the mapping. However, this also means that the referenced document is mutable and therefore changes throughout the mapping. For example, with the following Bloblang: ```bloblang root.rejected = this.invitees.filter(i -> i.mood < 0.5) root.invitees = this.invitees.filter(i -> i.mood >= 0.5) # In: {"invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Notice that we create a field `rejected` by copying the array field `invitees` and filtering out objects with a high mood. We then overwrite the field `invitees` by filtering out objects with a low mood, resulting in two array fields that are each a subset of the original. If we were to reverse the ordering of these assignments like so: ```bloblang root.invitees = this.invitees.filter(i -> i.mood >= 0.5) root.rejected = this.invitees.filter(i -> i.mood < 0.5) # In: {"invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Then the new field `rejected` would be empty as we have already mutated `invitees` to exclude the objects that it would be populated by. We can solve this problem either by carefully ordering our assignments or by capturing the original array using a variable (`let invitees = this.invitees`). Mutations are advantageous over a standard mapping in situations where the result is a document with mostly the same shape as the input document, since we can avoid unnecessarily copying data from the referenced input document. However, in situations where we are creating an entirely new document shape it can be more convenient to use the traditional [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) instead. ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the error is logged and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired [fallback behavior](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the documents down to just the ID and only those fans with an obsession score above 0.5, giving us: ```json { "id":"foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - mutation: | root.description = deleted() root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - mutation: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` --- # Page 258: nats_kv **URL**: https://docs.redpanda.com/connect/components/processors/nats_kv.md --- # nats_kv --- title: nats_kv latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/nats_kv page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/nats_kv.adoc description: Perform operations on a NATS key-value bucket. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs operations on a NATS key-value bucket. Introduced in version 4.13.0. ## [](#kv-operations)KV operations The NATS KV processor supports a multitude of KV operations via the [`operation`](#operation) field. Along with `get`, `put`, and `delete`, this processor supports atomic operations like `update` and `create`, as well as utility operations like `purge`, `history`, and `keys`. ## [](#metadata)Metadata This processor adds the following metadata fields to each message, depending on the chosen `operation`: ### [](#get-get_revision)get, get_revision - `nats_kv_key` - `nats_kv_bucket` - `nats_kv_revision` - `nats_kv_delta` - `nats_kv_operation` - `nats_kv_created` ### [](#create-update-delete-purge)create, update, delete, purge - `nats_kv_key` - `nats_kv_bucket` - `nats_kv_revision` - `nats_kv_operation` ### [](#keys)keys - `nats_kv_bucket` ## [](#connection-name)Connection name When monitoring and managing a production NATS system, it is often useful to know which connection a message was send/received from. This can be achieved by setting the connection name option when creating a NATS connection. Redpanda Connect will automatically set the connection name based off the label of the given NATS component, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication There are several components within Redpanda Connect which uses NATS services. You will find that each of these components support optional advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [User Credentials](https://docs.nats.io/using-nats/developer/connecting/creds). See an [in-depth tutorial](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkey-file)NKey file The NATS server can use these NKeys in several ways for authentication. The simplest is for the server to be configured with a list of known public keys and for the clients to respond to the challenge by signing it with its private NKey configured in the `nkey_file` or `nkey` field. [More details](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server supports decentralized authentication based on JSON Web Tokens (JWT). Clients need an [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) when connecting to a server which is configured to use this authentication scheme. The `user_credentials_file` field should point to a file containing both the private key and the JWT and can be generated with the [nsc tool](https://docs.nats.io/nats-tools/nsc). Alternatively, the `user_jwt` field can contain a plain text JWT and the `user_nkey_seed` field can contain the plain text NKey Seed. [More details](https://docs.nats.io/using-nats/developer/connecting/creds). ### [](#token)Token The `token` field can contain a plain text token string for [token-based authentication](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/tokens). ### [](#user-and-password)User and password The `user` and `password` fields can be used for [username/password authentication](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/username_password). #### Common ```yml processor: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) operation: "" # No default (required) key: "" # No default (required) ``` #### Advanced ```yml processor: label: "" nats_kv: urls: [] # No default (required) max_reconnects: 0 # No default (optional) bucket: "" # No default (required) operation: "" # No default (required) key: "" # No default (required) revision: "" # No default (optional) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.16.0 or later. **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#key)`key` The key for each message. Supports [wildcards](https://docs.nats.io/nats-concepts/subjects#wildcards) for the `history` and `keys` operations. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: foo # --- key: foo.bar.baz # --- key: foo.* # --- key: foo.> # --- key: foo.${! json("meta.type") } ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#operation)`operation` The operation to perform on the KV bucket. **Type**: `string` | Option | Summary | | --- | --- | | create | Adds the key/value pair if it does not exist. Returns an error if it already exists. | | delete | Deletes the key/value pair, but keeps historical values. | | get | Returns the latest value for key. | | get_revision | Returns the value of key for the specified revision. | | history | Returns historical values of key as an array of objects containing the following fields: key, value, bucket, revision, delta, operation, created. | | keys | Returns the keys in the bucket which match the keys_filter as an array of strings. | | purge | Deletes the key/value pair and all historical values. | | put | Places a new value for the key into the store. | | update | Updates the value for key only if the revision matches the latest revision. | ### [](#revision)`revision` The revision of the key to operate on. Used for `get_revision` and `update` operations. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: revision: "42" # --- revision: ${! @nats_kv_revision } ``` ### [](#timeout)`timeout` The maximum period to wait on an operation before aborting and returning an error. Requires version 4.27.0 or later. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 259: nats_request_reply **URL**: https://docs.redpanda.com/connect/components/processors/nats_request_reply.md --- # nats_request_reply --- title: nats_request_reply latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/nats_request_reply page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/nats_request_reply.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/nats_request_reply.adoc description: Sends a message to a NATS subject and expects a reply, from a NATS subscriber acting as a responder, back. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sends a message to a NATS subject and expects a reply back from a NATS subscriber acting as a responder. Introduced in version 4.27.0. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_subject` - `nats_sequence_stream` - `nats_sequence_consumer` - `nats_num_delivered` - `nats_num_pending` - `nats_domain` - `nats_timestamp_unix_nano` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production NATS system, it is often useful to know which connection a message was send/received from. This can be achieved by setting the connection name option when creating a NATS connection. Redpanda Connect will automatically set the connection name based off the label of the given NATS component, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication There are several components within Redpanda Connect which uses NATS services. You will find that each of these components support optional advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [User Credentials](https://docs.nats.io/using-nats/developer/connecting/creds). See an [in-depth tutorial](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkey-file)NKey file The NATS server can use these NKeys in several ways for authentication. The simplest is for the server to be configured with a list of known public keys and for the clients to respond to the challenge by signing it with its private NKey configured in the `nkey_file` or `nkey` field. [More details](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server supports decentralized authentication based on JSON Web Tokens (JWT). Clients need an [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) when connecting to a server which is configured to use this authentication scheme. The `user_credentials_file` field should point to a file containing both the private key and the JWT and can be generated with the [nsc tool](https://docs.nats.io/nats-tools/nsc). Alternatively, the `user_jwt` field can contain a plain text JWT and the `user_nkey_seed` field can contain the plain text NKey Seed. [More details](https://docs.nats.io/using-nats/developer/connecting/creds). ### [](#token)Token The `token` field can contain a plain text token string for [token-based authentication](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/tokens). ### [](#user-and-password)User and password The `user` and `password` fields can be used for [username/password authentication](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/username_password). #### Common ```yml processor: label: "" nats_request_reply: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] timeout: 3s ``` #### Advanced ```yml processor: label: "" nats_request_reply: urls: [] # No default (required) max_reconnects: 0 # No default (optional) subject: "" # No default (required) inbox_prefix: "" # No default (optional) headers: {} metadata: include_prefixes: [] include_patterns: [] timeout: 3s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed (private key) for NATS authentication. NKeys provide cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). Requires version 4.84.0 or later. **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inbox_prefix)`inbox_prefix` Set an explicit inbox prefix for the response subject **Type**: `string` ```yaml # Examples: inbox_prefix: _INBOX_joe ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. Requires version 4.56.0 or later. **Type**: `int` ### [](#metadata-2)`metadata` Specify which metadata values are added to messages as NATS headers, alongside any explicit `headers`. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` A subject to write to. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: ${! meta("kafka_topic") } # --- subject: foo.${! json("meta.type") } ``` ### [](#timeout)`timeout` A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as 300ms, -1.5h or 2h45m. Valid time units are ns, us (or µs), ms, s, m, h. **Type**: `string` **Default**: `3s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 260: noop **URL**: https://docs.redpanda.com/connect/components/processors/noop.md --- # noop --- title: noop latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/noop page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/noop.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/noop.adoc description: Noop is a processor that does nothing, the message passes through unchanged. Why? Sometimes doing nothing is the braver option. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Noop is a processor that does nothing, the message passes through unchanged. Why? Sometimes doing nothing is the braver option. ## Common ```yml processor: label: "" noop: {} ``` ## Advanced ```yml processor: label: "" noop: {} ``` --- # Page 261: ollama_chat **URL**: https://docs.redpanda.com/connect/components/processors/ollama_chat.md --- # ollama_chat --- title: ollama_chat latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/ollama_chat page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ollama_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ollama_chat.adoc description: Generates responses to messages in a chat conversation, using the Ollama API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates responses to messages in a chat conversation using the Ollama API and external tools. Introduced in version 4.32.0. #### Common ```yml processor: label: "" ollama_chat: model: "" # No default (required) prompt: "" # No default (optional) image: "" # No default (optional) response_format: text max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) save_prompt_metadata: false history: "" # No default (optional) tools: [] runner: context_size: 0 # No default (optional) batch_size: 0 # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" ollama_chat: model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) image: "" # No default (optional) response_format: text max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) num_keep: 0 # No default (optional) seed: 0 # No default (optional) top_k: 0 # No default (optional) top_p: 0 # No default (optional) repeat_penalty: 0 # No default (optional) presence_penalty: 0 # No default (optional) frequency_penalty: 0 # No default (optional) stop: [] # No default (optional) save_prompt_metadata: false history: "" # No default (optional) max_tool_calls: 3 tools: [] runner: context_size: 0 # No default (optional) batch_size: 0 # No default (optional) gpu_layers: 0 # No default (optional) threads: 0 # No default (optional) use_mmap: false # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor sends prompts to your chosen Ollama large language model (LLM) and generates text from the responses using the Ollama API and external tools. By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) and [examples](#examples). ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set, download the Ollama binary to this directory and use it as a model cache. Requires version 4.34.0 or later. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set, download the Ollama binary from this URL. The default value is the official Ollama GitHub release for this platform. Requires version 4.34.0 or later. **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` A number between `-2.0` and `2.0` that the processor sends to the Ollama server as the `frequency_penalty` model option. Positive values penalize each token in proportion to how often it has already appeared in the text, which makes the model less likely to repeat the same line verbatim. A value of `0` is not sent, so the server’s default applies. Requires version 4.34.0 or later. **Type**: `float` ### [](#history)`history` Include historical messages in a chat request. You must use a Bloblang query to create an array of objects in the form of `[{"role": "", "content":""}]` where: - `role` is the sender of the original messages, either `system`, `user`, `assistant`, or `tool`. - `content` is the text of the original messages. Requires version 4.51.0 or later. **Type**: `string` ### [](#image)`image` An optional image to submit along with the `prompt` value. The result should be a byte array. Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: image: 'root = this.image.decode("base64") # decode base64 encoded image' ``` ### [](#max_tokens)`max_tokens` The maximum number of tokens to predict and output. Limiting the amount of output means that requests are processed faster and have a fixed limit on the cost. Requires version 4.34.0 or later. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of sequential calls you can make to external tools to retrieve additional information to answer a prompt. Requires version 4.45.0 or later. **Type**: `int` **Default**: `3` ### [](#model)`model` The name of the Ollama model to use. For a full list of models, see the [Ollama website](https://ollama.com/models). **Type**: `string` ```yaml # Examples: model: llama3.1 # --- model: gemma2 # --- model: qwen2 # --- model: phi3 ``` ### [](#num_keep)`num_keep` Specify the number of tokens from the initial prompt to retain when the model resets its internal context. By default, this value is set to `4`. Use `-1` to retain all tokens from the initial prompt. Requires version 4.34.0 or later. **Type**: `int` ### [](#presence_penalty)`presence_penalty` A number between `-2.0` and `2.0` that the processor sends to the Ollama server as the `presence_penalty` model option. Positive values penalize every token that has already appeared in the text, regardless of how often, which makes the model more likely to move on to new topics. A value of `0` is not sent, so the server’s default applies. Requires version 4.34.0 or later. **Type**: `float` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#repeat_penalty)`repeat_penalty` Sets how strongly to penalize repetitions. A higher value, for example 1.5, will penalize repetitions more strongly. A lower value, for example 0.9, will be more lenient. Requires version 4.34.0 or later. **Type**: `float` ### [](#response_format)`response_format` The format of the response the Ollama model generates. If specifying JSON output, then the `prompt` should specify that the output should be in JSON as well. Requires version 4.34.0 or later. **Type**: `string` **Default**: `text` **Options**: `text`, `json` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. Requires version 4.34.0 or later. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` Sets the number of layers to offload to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Sets the number of threads to use during response generation. For optimal performance, set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. Set to `true` to load only the necessary parts of the model into memory. This setting is only supported on Unix systems. **Type**: `bool` ### [](#save_prompt_metadata)`save_prompt_metadata` Set to `true` to save the prompt value to a metadata field (`@prompt`) on the corresponding output message. If you use the `system_prompt` field, its value is also saved to an `@system_prompt` metadata field on each output message. Requires version 4.42.0 or later. **Type**: `bool` **Default**: `false` ### [](#seed)`seed` Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. Requires version 4.34.0 or later. **Type**: `int` ```yaml # Examples: seed: 42 ``` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave this field blank and the processor starts and runs a local Ollama server, or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ### [](#stop)`stop[]` Sets the stop sequences to use. When this pattern is encountered, the LLM stops generating text and returns the final response. Requires version 4.34.0 or later. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` The temperature of the model. Increasing the temperature makes the model answer more creatively. Requires version 4.34.0 or later. **Type**: `int` ### [](#tools)`tools[]` The external tools the LLM can invoke, such as functions, APIs, or web browsing. You can build subpipelines of processors that include definitions of these tools, and the specified LLM can choose when to invoke them to help answer a prompt. Requires version 4.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool. The LLM uses it to decide whether to invoke the tool. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The parameters the LLM can provide when it invokes this tool, keyed by parameter name. **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` The values this parameter is limited to. Leave empty to accept any value. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The names of the parameters the LLM must provide when it invokes this tool. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The processors to run when the LLM invokes this tool. They receive a message whose payload is the tool call arguments as a JSON object, and their output is returned to the LLM as the tool result. **Type**: `array` ### [](#top_k)`top_k` Reduces the probability of generating nonsense. A higher value, for example `100`, will give more diverse answers. A lower value, for example `10`, will be more conservative. Requires version 4.34.0 or later. **Type**: `int` ### [](#top_p)`top_p` Works together with `top-k`. A higher value, for example 0.95, will lead to more diverse text. A lower value, for example 0.5, will generate more focused and conservative text. Requires version 4.34.0 or later. **Type**: `float` ## [](#examples)Examples ### [](#use-llava-to-analyze-an-image)Use Llava to analyze an image This example fetches image URLs from stdin and has a multimodal LLM describe the image. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - http: verb: GET url: "${!content().string()}" - ollama_chat: model: llava prompt: "Describe the following image" image: "root = content()" output: stdout: codec: lines ``` ### [](#use-subpipelines-as-tool-calls)Use subpipelines as tool calls This example allows llama3.2 to execute a subpipeline as a tool call to get more data. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - ollama_chat: model: llama3.2 prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to lookup the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: # Spoof curl user-ageent to get a plaintext text User-Agent: curl/8.11.1 output: stdout: {} ``` --- # Page 262: ollama_embeddings **URL**: https://docs.redpanda.com/connect/components/processors/ollama_embeddings.md --- # ollama_embeddings --- title: ollama_embeddings latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/ollama_embeddings page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ollama_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ollama_embeddings.adoc description: Generates vector embeddings from text, using the Ollama API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates vector embeddings from text, using the Ollama API. Introduced in version 4.32.0. This processor sends text to your chosen Ollama large language model (LLM) and creates vector embeddings, using the Ollama API. Vector embeddings are long arrays of numbers that represent values or objects, in this case text. By default, the processor starts and runs a locally installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs). #### Common ```yml processor: label: "" ollama_embeddings: model: "" # No default (required) text: "" # No default (optional) runner: context_size: 0 # No default (optional) batch_size: 0 # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" ollama_embeddings: model: "" # No default (required) text: "" # No default (optional) runner: context_size: 0 # No default (optional) batch_size: 0 # No default (optional) gpu_layers: 0 # No default (optional) threads: 0 # No default (optional) use_mmap: false # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set, download the Ollama binary to this directory and use it as a model cache. Requires version 4.34.0 or later. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set, download the Ollama binary from this URL. The default value is the official Ollama GitHub release for this platform. Requires version 4.34.0 or later. **Type**: `string` ### [](#model)`model` The name of the Ollama model to use. For a full list of models, see the [Ollama website](https://ollama.com/models). **Type**: `string` ```yaml # Examples: model: nomic-embed-text # --- model: mxbai-embed-large # --- model: snowflake-artic-embed # --- model: all-minilm ``` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. Requires version 4.34.0 or later. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` Sets the number of layers to offload to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Sets the number of threads to use during response generation. For optimal performance, set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. Set to `true` to load only the necessary parts of the model into memory. This setting is only supported on Unix systems. **Type**: `bool` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave this field blank and the processor starts and runs a local Ollama server, or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ### [](#text)`text` The text you want to generate vector embeddings for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ## [](#examples)Examples ### [](#store-embedding-vectors-in-qdrant)Store embedding vectors in Qdrant Computes embeddings for some generated data and stores them in [Qdrant](https://docs.redpanda.com/connect/components/outputs/qdrant/). ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - ollama_embeddings: model: snowflake-artic-embed text: "${!this.text}" output: qdrant: grpc_host: localhost:6334 collection_name: "example_collection" id: "root = uuid_v4()" vector_mapping: "root = this" ``` ### [](#store-embedding-vectors-in-cyborgdb)Store embedding vectors in CyborgDB Computes embeddings for some generated data and stores them in [CyborgDB](https://docs.redpanda.com/connect/components/outputs/cyborgdb/). ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - ollama_embeddings: model: snowflake-artic-embed text: "${!this.text}" output: cyborgdb: host: "${CYBORGDB_HOST}" api_key: "${CYBORGDB_API_KEY}" index_key: "${CYBORGDB_INDEX_KEY}" index_name: "my_encrypted_index" operation: "upsert" id: "root = uuid_v4()" vector_mapping: "root = this" ``` ### [](#store-embedding-vectors-in-clickhouse)Store embedding vectors in Clickhouse Compute embeddings for some generated data and store it within [Clickhouse](https://clickhouse.com/) ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - branch: processors: - ollama_embeddings: model: snowflake-artic-embed text: "${!this.text}" result_map: | root.embeddings = this output: sql_insert: driver: clickhouse dsn: "clickhouse://localhost:9000" table: searchable_text columns: ["id", "text", "vector"] args_mapping: "root = [uuid_v4(), this.text, this.embeddings]" ``` --- # Page 263: ollama_moderation **URL**: https://docs.redpanda.com/connect/components/processors/ollama_moderation.md --- # ollama_moderation --- title: ollama_moderation latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/ollama_moderation page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ollama_moderation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ollama_moderation.adoc description: Checks whether responses from a large language model (LLM) are safe, using the Ollama API. page-git-created-date: "2025-01-28" page-git-modified-date: "2026-10-10" --- Generates responses to messages in a chat conversation using the Ollama API, and checks the responses to make sure they do not violate [safety or security standards](https://mlcommons.org/2024/04/mlc-aisafety-v0-5-poc/). Introduced in version 4.45.0. #### Common ```yml processor: label: "" ollama_moderation: model: "" # No default (required) prompt: "" # No default (required) response: "" # No default (required) runner: context_size: 0 # No default (optional) batch_size: 0 # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" ollama_moderation: model: "" # No default (required) prompt: "" # No default (required) response: "" # No default (required) runner: context_size: 0 # No default (optional) batch_size: 0 # No default (optional) gpu_layers: 0 # No default (optional) threads: 0 # No default (optional) use_mmap: false # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor checks the safety of responses from your chosen large language model (LLM) using either [Llama Guard 3](https://ollama.com/library/llama-guard3) or [ShieldGemma](https://ollama.com/library/shieldgemma). By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) and [Examples](#examples). To check the safety of your prompts, see the [`ollama_chat` processor](https://docs.redpanda.com/connect/components/processors/ollama_chat/#examples) documentation. ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set, download the Ollama binary to this directory and use it as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set, download the Ollama binary from this URL. The default value is the official Ollama GitHub release for this platform. **Type**: `string` ### [](#model)`model` The name of the Ollama LLM to use. **Type**: `string` | Option | Summary | | --- | --- | | llama-guard3 | When using llama-guard3, two pieces of metadata is added: @safe with the value of yes or no and the second being @category for the safety category violation. For more information see the Llama Guard 3 Model Card. | | shieldgemma | When using shieldgemma, the model output is a single piece of metadata of @safe with a value of yes or no if the response is not in violation of its defined safety policies. | ```yaml # Examples: model: llama-guard3 # --- model: shieldgemma ``` ### [](#prompt)`prompt` The prompt you used to generate a response from an LLM. If you’re using the `ollama_chat` processor, you can set the `save_prompt_metadata` field to save the contents of your prompts. You can then run them through the `ollama_moderation` processor to check the model responses for safety. You can also check the safety of your prompts. For more information, see the [`ollama_chat` processor](https://docs.redpanda.com/connect/components/processors/ollama_chat/#examples) documentation. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response)`response` The LLM’s response that you want to check for safety (classify as safe or unsafe content). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` Sets the number of layers to offload to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Sets the number of threads to use during response generation. For optimal performance, set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. Set to `true` to load only the necessary parts of the model into memory. This setting is only supported on Unix systems. **Type**: `bool` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave this field blank and the processor starts and runs a local Ollama server, or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ## [](#examples)Examples ### [](#use-llama-guard-3-classify-a-llm-response)Use Llama Guard 3 classify a LLM response This example uses Llama Guard 3 to check if another model responded with a safe or unsafe content. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - ollama_chat: model: llava prompt: "${!content().string()}" save_prompt_metadata: true - ollama_moderation: model: llama-guard3 prompt: "${!@prompt}" response: "${!content().string()}" - mapping: | root.response = content().string() root.is_safe = @safe output: stdout: codec: lines ``` --- # Page 264: openai_chat_completion **URL**: https://docs.redpanda.com/connect/components/processors/openai_chat_completion.md --- # openai_chat_completion --- title: openai_chat_completion latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/openai_chat_completion page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_chat_completion.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_chat_completion.adoc description: Generates responses to messages in a chat conversation, using the OpenAI API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates responses to messages in a chat conversation, using the OpenAI API and external tools. Introduced in version 4.32.0. #### Common ```yml processor: label: "" openai_chat_completion: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) image: "" # No default (optional) max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) user: "" # No default (optional) response_format: text json_schema: name: "" # No default (required) schema: "" # No default (required) tools: [] # No default (required) ``` #### Advanced ```yml processor: label: "" openai_chat_completion: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) image: "" # No default (optional) max_tokens: 0 # No default (optional) temperature: 0 # No default (optional) user: "" # No default (optional) response_format: text json_schema: name: "" # No default (required) description: "" # No default (optional) schema: "" # No default (required) schema_registry: url: "" # No default (required) name_prefix: schema_registry_id_ subject: "" # No default (required) refresh_interval: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} top_p: 0 # No default (optional) frequency_penalty: 0 # No default (optional) presence_penalty: 0 # No default (optional) seed: 0 # No default (optional) stop: [] # No default (optional) tools: [] # No default (required) ``` This processor sends user prompts to the OpenAI API, and the specified large language model (LLM) generates responses using all available context, including supplementary data provided by [external tools](#tools). By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` configuration field to customize it. To learn more about chat completion, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/chat-completions), and [Examples](#examples). ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Specify a number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. Requires version 4.34.0 or later. **Type**: `float` ### [](#history)`history` Include messages from a prior conversation. You must use a Bloblang query to create an array of objects in the form of `[{"role": "user", "content": ""}, {"role":"assistant", "content":""}]` where: - `role` is the sender of the original messages, either `system`, `user`, or `assistant`. - `content` is the text of the original messages. Requires version 4.51.0 or later. **Type**: `string` ### [](#image)`image` An optional image to submit along with the prompt. The result of the Bloblang mapping must be a byte array. Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: image: 'root = this.image.decode("base64") # decode base64 encoded image' ``` ### [](#json_schema)`json_schema` The JSON schema used by the model when generating responses in `json_schema` format. To learn more about supported JSON schema features, see the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas). Requires version 4.35.0 or later. **Type**: `object` ### [](#json_schema-description)`json_schema.description` An optional description, which helps the model understand the schema’s purpose. **Type**: `string` ### [](#json_schema-name)`json_schema.name` The name of the JSON schema to use. **Type**: `string` ### [](#json_schema-schema)`json_schema.schema` The JSON schema for the model to use when generating the output. **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to generate for chat completion. Requires version 4.34.0 or later. **Type**: `int` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: gpt-4o # --- model: gpt-4o-mini # --- model: gpt-4 # --- model: gpt4-turbo ``` ### [](#presence_penalty)`presence_penalty` Specify a number between `-2.0` and `2.0`. Positive values penalize new tokens if they have appeared in the text so far. This increases the model’s likelihood to talk about new topics. Requires version 4.34.0 or later. **Type**: `float` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload of each message as a string. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` Specify the output format of the configured `model`. If you choose the `json_schema` option, you must also configure a `json_schema` or `schema_registry`. Requires version 4.35.0 or later. **Type**: `string` **Default**: `text` **Options**: `text`, `json`, `json_schema` ### [](#schema_registry)`schema_registry` The schema registry to dynamically load schemas for model responses in `json_schema` format. Schemas must be in JSON format. To learn more about supported JSON schema features, see the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas). Requires version 4.35.0 or later. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-name_prefix)`schema_registry.name_prefix` A prefix to add to the schema registry name. To form the complete schema registry name, the schema ID is appended as a suffix. **Type**: `string` **Default**: `schema_registry_id_` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-refresh_interval)`schema_registry.refresh_interval` How frequently to poll the schema registry for the latest schema. If not specified, the schema does not refresh. **Type**: `string` ### [](#schema_registry-subject)`schema_registry.subject` The subject name used to fetch the schema from the schema registry. **Type**: `string` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed)`seed` When set to a specific number, the model makes a best effort to generate consistent responses for requests that use the same prompt, seed, and parameters. Determinism is not guaranteed. Requires version 4.34.0 or later. **Type**: `int` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#stop)`stop[]` Specify up to four stop sequences to use. When the model encounters a stop pattern, it stops generating text and returns the final response. Requires version 4.34.0 or later. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Choose a sampling temperature between `0` and `2`: - Higher values, such as `0.8` make the output more random. - Lower values, such as `0.2` make the output more focused and deterministic. Redpanda recommends adding a value for this field or `top_p`, but not both. Requires version 4.34.0 or later. **Type**: `float` ### [](#tools)`tools[]` External tools the model can invoke, such as functions, APIs, or web browsing. You can build subpipelines of processors that include definitions of these tools, and the specified model can choose when to invoke them to help answer a prompt. > 📝 **NOTE** > > If you don’t want to use external tools, enter an empty array `tools: []`. Requires version 4.50.0 or later. **Type**: `array` ### [](#tools-description)`tools[].description` A description of this tool. The LLM uses it to decide whether to invoke the tool. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` **Default**: `[]` ### [](#tools-parameters-properties)`tools[].parameters.properties` The parameters the LLM can provide when it invokes this tool, keyed by parameter name. **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` The values this parameter is limited to. Leave empty to accept any value. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The names of the parameters the LLM must provide when it invokes this tool. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The processors to run when the LLM invokes this tool. They receive a message whose payload is the tool call arguments as a JSON object, and their output is returned to the LLM as the tool result. **Type**: `array` ### [](#top_p)`top_p` An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with `top_p` probability mass. For example, a `top_p` of `0.1` means only the tokens comprising the top 10% probability mass are sampled. Redpanda recommends adding a value for this field or `temperature`, but not both. Requires version 4.34.0 or later. **Type**: `float` ### [](#user)`user` A unique identifier that represents the end-user generating the prompt. This value can help OpenAI monitor and detect [platform abuse](https://openai.com/policies/usage-policies/). This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.34.0 or later. **Type**: `string` ## [](#examples)Examples ### [](#use-gpt-4o-analyze-an-image)Use GPT-4o analyze an image This example fetches image URLs from stdin and has GPT-4o describe the image. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - http: verb: GET url: "${!content().string()}" - openai_chat_completion: model: gpt-4o api_key: TODO prompt: "Describe the following image" image: "root = content()" output: stdout: codec: lines ``` ### [](#provide-historical-chat-history)Provide historical chat history This pipeline provides a historical chat history to GPT-4o using a cache. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - mapping: | root.prompt = content().string() - branch: processors: - cache: resource: mem operator: get key: history - catch: - mapping: 'root = []' result_map: 'root.history = this' - branch: processors: - openai_chat_completion: model: gpt-4o api_key: TODO prompt: "${!this.prompt}" history: 'root = this.history' result_map: 'root.response = content().string()' - mutation: | root.history = this.history.concat([ {"role": "user", "content": this.prompt}, {"role": "assistant", "content": this.response}, ]) - cache: resource: mem operator: set key: history value: '${!this.history}' - mapping: | root = this.response output: stdout: codec: lines cache_resources: - label: mem memory: {} ``` ### [](#use-gpt-4o-to-call-a-tool)Use GPT-4o to call a tool This example asks GPT-4o to respond with the weather by invoking an HTTP processor to get the forecast. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - openai_chat_completion: model: gpt-4o api_key: "${OPENAI_API_KEY}" prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to look up the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: User-Agent: curl/8.11.1 # Returns a text string from the weather website output: stdout: {} ``` --- # Page 265: openai_embeddings **URL**: https://docs.redpanda.com/connect/components/processors/openai_embeddings.md --- # openai_embeddings --- title: openai_embeddings latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/openai_embeddings page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_embeddings.adoc description: Generates vector embeddings to represent input text, using the OpenAI API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates vector embeddings to represent input text, using the OpenAI API. Introduced in version 4.32.0. This processor sends text strings to the OpenAI API, which generates vector embeddings. By default, the processor submits the entire payload of each message as a string, unless you use the `text_mapping` configuration field to customize it. To learn more about vector embeddings, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/embeddings). #### Common ```yml processor: label: "" openai_embeddings: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) text_mapping: "" # No default (optional) dimensions: 0 # No default (optional) ``` #### Advanced ```yml processor: label: "" openai_embeddings: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) text_mapping: "" # No default (optional) dimensions: 0 # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#dimensions)`dimensions` The number of dimensions the resulting output embeddings should have. Only supported in `text-embedding-3` and later models. Requires version 4.34.0 or later. **Type**: `int` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: text-embedding-3-large # --- model: text-embedding-3-small # --- model: text-embedding-ada-002 ``` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#text_mapping)`text_mapping` A Bloblang mapping that returns the text you want to generate vector embeddings for. By default, the processor submits the entire payload of each message as a string. **Type**: `string` ## [](#examples)Examples ### [](#store-embedding-vectors-in-pinecone)Store embedding vectors in Pinecone Computes embeddings for some generated data and stores them in [Pinecone](https://docs.redpanda.com/connect/components/outputs/pinecone/). ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - openai_embeddings: model: text-embedding-3-large api_key: "${OPENAI_API_KEY}" text_mapping: "root = this.text" output: pinecone: host: "${PINECONE_HOST}" api_key: "${PINECONE_API_KEY}" id: "root = uuid_v4()" vector_mapping: "root = this" ``` ### [](#store-embedding-vectors-in-cyborgdb)Store embedding vectors in CyborgDB Computes embeddings for some generated data and stores them in [CyborgDB](https://docs.redpanda.com/connect/components/outputs/cyborgdb/). ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - openai_embeddings: model: text-embedding-3-large api_key: "${OPENAI_API_KEY}" text_mapping: "root = this.text" output: cyborgdb: host: "${CYBORGDB_HOST}" api_key: "${CYBORGDB_API_KEY}" index_key: "${CYBORGDB_INDEX_KEY}" index_name: "my_encrypted_index" operation: "upsert" id: "root = uuid_v4()" vector_mapping: "root = this" ``` --- # Page 266: openai_image_generation **URL**: https://docs.redpanda.com/connect/components/processors/openai_image_generation.md --- # openai_image_generation --- title: openai_image_generation latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/openai_image_generation page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_image_generation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_image_generation.adoc description: Generates an image from a text description and other attributes, using OpenAI API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates an image from a text description and other attributes, using OpenAI API. Introduced in version 4.32.0. This processor sends an image description and other attributes, such as image size and quality to the OpenAI API, which generates an image. By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` configuration field to customize it. To learn more about image generation, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/images). #### Common ```yml processor: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) quality: "" # No default (optional) size: "" # No default (optional) style: "" # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: dall-e-3 # --- model: dall-e-2 ``` ### [](#prompt)`prompt` A text description of the image you want to generate. The `prompt` field accepts a maximum of 1000 characters for `dall-e-2` and 4000 characters for `dall-e-3`. **Type**: `string` ### [](#quality)`quality` The quality of the image to generate. Use `hd` to create images with finer details and greater consistency across the image. This parameter is only supported for `dall-e-3` models. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: quality: standard # --- quality: hd ``` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#size)`size` The size of the generated image. Choose from `256x256`, `512x512`, or `1024x1024` for `dall-e-2`. Choose from `1024x1024`, `1792x1024`, or `1024x1792` for `dall-e-3` models. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: size: 1024x1024 # --- size: 512x512 # --- size: 1792x1024 # --- size: 1024x1792 ``` ### [](#style)`style` The style of the generated image. Choose from `vivid` or `natural`. Vivid causes the model to lean towards generating hyperreal and dramatic images. Natural causes the model to produce more natural, less hyperreal looking images. This parameter is only supported for `dall-e-3`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: style: vivid # --- style: natural ``` --- # Page 267: openai_speech **URL**: https://docs.redpanda.com/connect/components/processors/openai_speech.md --- # openai_speech --- title: openai_speech latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/openai_speech page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_speech.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_speech.adoc description: Generates audio from a text description and other attributes, using OpenAI API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates audio from a text description and other attributes, using OpenAI API. Introduced in version 4.32.0. This processor sends a text description and other attributes, such as a voice type and format to the OpenAI API, which generates audio. By default, the processor submits the entire payload of each message as a string, unless you use the `input` configuration field to customize it. To learn more about turning text into spoken audio, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/text-to-speech). #### Common ```yml processor: label: "" openai_speech: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) input: "" # No default (optional) voice: "" # No default (required) ``` #### Advanced ```yml processor: label: "" openai_speech: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) input: "" # No default (optional) voice: "" # No default (required) response_format: "" # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#input)`input` A text description of the audio you want to generate. The `input` field accepts a maximum of 4096 characters. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: tts-1 # --- model: tts-1-hd ``` ### [](#response_format)`response_format` The format to generate audio in. Default is `mp3`. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: response_format: mp3 # --- response_format: opus # --- response_format: aac # --- response_format: flac # --- response_format: wav # --- response_format: pcm ``` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#voice)`voice` The type of voice to use when generating the audio. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: voice: alloy # --- voice: echo # --- voice: fable # --- voice: onyx # --- voice: nova # --- voice: shimmer ``` --- # Page 268: openai_transcription **URL**: https://docs.redpanda.com/connect/components/processors/openai_transcription.md --- # openai_transcription --- title: openai_transcription latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/openai_transcription page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_transcription.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_transcription.adoc description: Generates a transcription of spoken audio in the input language, using the OpenAI API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Generates a transcription of spoken audio in the input language, using the OpenAI API. Introduced in version 4.32.0. This processor sends an audio file object along with the input language to OpenAI API to generate a transcription. By default, the processor submits the entire payload of each message as a string, unless you use the `file` configuration field to customize it. To learn more about audio transcription, see the: [OpenAI API documentation](https://platform.openai.com/docs/guides/speech-to-text). #### Common ```yml processor: label: "" openai_transcription: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (required) ``` #### Advanced ```yml processor: label: "" openai_transcription: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (required) language: "" # No default (optional) prompt: "" # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#file)`file` The audio file object (not file name) to transcribe, in one of the following formats: `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm`. **Type**: `string` ### [](#language)`language` The language of the input audio. Supplying the input language in ISO-639-1 format improves accuracy and latency. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: language: en # --- language: fr # --- language: de # --- language: zh ``` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: whisper-1 ``` ### [](#prompt)`prompt` Optional text to guide the model’s style or continue a previous audio segment. The prompt should match the audio language. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` --- # Page 269: openai_translation **URL**: https://docs.redpanda.com/connect/components/processors/openai_translation.md --- # openai_translation --- title: openai_translation latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/openai_translation page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_translation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_translation.adoc description: Translates spoken audio into English, using the OpenAI API. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Translates spoken audio into English, using the OpenAI API. Introduced in version 4.32.0. This processor sends an audio file object to OpenAI API to generate a translation. By default, the processor submits the entire payload of each message as a string, unless you use the `file` configuration field to customize it. To learn more about translation, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/speech-to-text). #### Common ```yml processor: label: "" openai_translation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" openai_translation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (optional) prompt: "" # No default (optional) ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#file)`file` The audio file object (not file name) to translate, in one of the following formats: `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm`. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: whisper-1 ``` ### [](#prompt)`prompt` Optional text to guide the model’s style or continue a previous audio segment. The prompt should match the audio language. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` --- # Page 270: parallel **URL**: https://docs.redpanda.com/connect/components/processors/parallel.md --- # parallel --- title: parallel latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/parallel page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parallel.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parallel.adoc description: A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the for_each processor), but where each message is processed in parallel. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the [`for_each`](https://docs.redpanda.com/connect/components/processors/for_each/) processor), but where each message is processed in parallel. The field `cap`, if greater than zero, caps the maximum number of parallel processing threads. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching in [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). #### Common ```yml processor: label: "" parallel: cap: 0 processors: [] # No default (required) ``` #### Advanced ```yml processor: label: "" parallel: cap: 0 processors: [] # No default (required) ``` ## [](#fields)Fields ### [](#cap)`cap` The maximum number of messages to have processing at a given time. **Type**: `int` **Default**: `0` ### [](#processors)`processors[]` A list of child processors to apply. **Type**: `array` --- # Page 271: parquet_decode **URL**: https://docs.redpanda.com/connect/components/processors/parquet_decode.md --- # parquet_decode --- title: parquet_decode latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/parquet_decode page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parquet_decode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parquet_decode.adoc description: Decodes Parquet files into a batch of structured messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Decodes [Parquet files](https://parquet.apache.org/docs/) into a batch of structured messages. Introduced in version 4.4.0. This processor uses [https://github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go), which is itself experimental. As a result, how this processor functions can change outside of major version releases. #### Common ```yml processor: label: "" parquet_decode: handle_logical_types: v1 ``` #### Advanced ```yml processor: label: "" parquet_decode: handle_logical_types: v1 ``` ## [](#fields)Fields ### [](#handle_logical_types)`handle_logical_types` Set to `v2` to enable enhanced decoding of logical types, or keep the default value (`v1`) to ignore logical type metadata when decoding values. In Parquet format, logical types are represented using standard physical types along with metadata that provides additional context. For example, UUIDs are stored as a `FIXED_LEN_BYTE_ARRAY` physical type, but the schema metadata identifies them as UUIDs. By enabling `v2`, this processor uses the metadata descriptions of logical types to produce more meaningful values during decoding. > 📝 **NOTE** > > Each version enables the logical type handling for that version and all earlier versions, which allows handling of new logical types to be introduced without breaking existing pipelines. When creating new pipelines, Redpanda recommends that you use the newest available version. Requires version 4.51.0 or later. **Type**: `string` **Default**: `v1` | Option | Summary | | --- | --- | | v1 | No special handling of logical types | | v2 | TIMESTAMP - decodes as an RFC3339 string describing the time. If the isAdjustedToUTC flag is set to true in the parquet file, the time zone will be set to UTC. If it is set to false the time zone will be set to local time.UUID - decodes as a string, i.e. 00112233-4455-6677-8899-aabbccddeeff. | ```yaml # Examples: handle_logical_types: v2 ``` ## [](#examples)Examples ### [](#reading-parquet-files-from-aws-s3)Reading Parquet Files from AWS S3 In this example we consume files from AWS S3 as they’re written by listening onto an SQS queue for upload events. We make sure to use the `to_the_end` scanner which means files are read into memory in full, which then allows us to use a `parquet_decode` processor to expand each file into a batch of messages. Finally, we write the data out to local files as newline delimited JSON. ```yaml input: aws_s3: bucket: TODO prefix: foos/ scanner: to_the_end: {} sqs: url: TODO processors: - parquet_decode: {} output: file: codec: lines path: './foos/${! meta("s3_key") }.jsonl' ``` --- # Page 272: parquet_encode **URL**: https://docs.redpanda.com/connect/components/processors/parquet_encode.md --- # parquet_encode --- title: parquet_encode latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/parquet_encode page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parquet_encode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parquet_encode.adoc description: Encodes Parquet files from a batch of structured messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Encodes [Parquet files](https://parquet.apache.org/docs/) from a batch of structured messages. Introduced in version 4.4.0. This processor uses [https://github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go), which is itself experimental. As a result, how this processor functions can change outside of major version releases. #### Common ```yml processor: label: "" parquet_encode: schema: [] # No default (optional) schema_metadata: "" default_compression: uncompressed ``` #### Advanced ```yml processor: label: "" parquet_encode: schema: [] # No default (optional) schema_metadata: "" default_compression: uncompressed default_encoding: DELTA_LENGTH_BYTE_ARRAY default_timestamp_unit: NANOSECOND ``` ## [](#fields)Fields ### [](#default_compression)`default_compression` The default compression type to use for fields. **Type**: `string` **Default**: `uncompressed` **Options**: `uncompressed`, `snappy`, `gzip`, `brotli`, `zstd`, `lz4raw` ### [](#default_encoding)`default_encoding` The default encoding type to use for fields. A custom default encoding is only necessary when consuming data with libraries that do not support `DELTA_LENGTH_BYTE_ARRAY`. Requires version 4.11.0 or later. **Type**: `string` **Default**: `DELTA_LENGTH_BYTE_ARRAY` **Options**: `DELTA_LENGTH_BYTE_ARRAY`, `PLAIN` ### [](#default_timestamp_unit)`default_timestamp_unit` The precision used when encoding TIMESTAMP logical types. The default `NANOSECOND` matches historical behaviour, but `TIMESTAMP(NANOS)` is not readable by Apache Spark (Databricks), AWS Athena or DuckDB; set this to `MICROSECOND` (or `MILLISECOND`) when writing Parquet files intended for consumption by those engines. Requires version 4.89.0 or later. **Type**: `string` **Default**: `NANOSECOND` **Options**: `NANOSECOND`, `MICROSECOND`, `MILLISECOND` ### [](#schema)`schema[]` Parquet schema. **Type**: `array` ### [](#schema-fields)`schema[].fields[]` A list of child fields. **Type**: `array` ```yaml # Examples: fields: - name: foo type: INT64 - name: bar type: BYTE_ARRAY ``` ### [](#schema-name)`schema[].name` The name of the column. **Type**: `string` ### [](#schema-optional)`schema[].optional` Whether the field is optional. **Type**: `bool` **Default**: `false` ### [](#schema-repeated)`schema[].repeated` Whether the field is repeated. **Type**: `bool` **Default**: `false` ### [](#schema-type)`schema[].type` The type of the column, only applicable for leaf columns with no child fields. Some logical types can be specified here such as UTF8. **Type**: `string` **Options**: `BOOLEAN`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BYTE_ARRAY`, `UTF8`, `TIMESTAMP`, `BSON`, `ENUM`, `JSON`, `UUID` ### [](#schema_metadata)`schema_metadata` Optionally specify a metadata field containing a schema definition to use for encoding instead of a statically defined schema. For batches of messages, the first message’s schema will be applied to all subsequent messages of the batch. Requires version 4.62.0 or later. **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#writing-parquet-files-to-aws-s3)Writing Parquet Files to AWS S3 In this example we use the batching mechanism of an `aws_s3` output to collect a batch of messages in memory, which then converts it to a parquet file and uploads it. ```yaml output: aws_s3: bucket: TODO path: 'stuff/${! timestamp_unix() }-${! uuid_v4() }.parquet' batching: count: 1000 period: 10s processors: - parquet_encode: schema: - name: id type: INT64 - name: weight type: DOUBLE - name: content type: BYTE_ARRAY default_compression: zstd ``` --- # Page 273: parquet **URL**: https://docs.redpanda.com/connect/components/processors/parquet.md --- # parquet --- title: parquet latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/parquet page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parquet.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parquet.adoc description: Converts batches of documents to or from Parquet files. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **WARNING: Deprecated** > > Deprecated > > This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). Converts batches of documents to or from [Parquet files](https://parquet.apache.org/docs/). Introduced in version 3.62.0. #### Common ```yml processor: label: "" parquet: operator: "" # No default (required) compression: snappy schema_file: "" # No default (optional) schema: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" parquet: operator: "" # No default (required) compression: snappy schema_file: "" # No default (optional) schema: "" # No default (optional) ``` ## [](#alternatives)Alternatives This processor is now deprecated, it’s recommended that you use the new [`parquet_decode`](https://docs.redpanda.com/connect/components/processors/parquet_decode/) and [`parquet_encode`](https://docs.redpanda.com/connect/components/processors/parquet_encode/) processors as they provide a number of advantages, the most important of which is better error messages for when schemas are mismatched or files could not be consumed. ## [](#troubleshooting)Troubleshooting This processor is experimental and the error messages that it provides are often vague and unhelpful. An error message of the form `interface \{} is nil, not ` implies that a field of the given type was expected but not found in the processed message when writing parquet files. Unfortunately the name of the field will sometimes be missing from the error, in which case it’s worth double checking the schema you provided to make sure that there are no typos in the field names, and if that doesn’t reveal the issue it can help to mark fields as OPTIONAL in the schema and gradually change them back to REQUIRED until the error returns. ## [](#define-the-schema)Define the schema The schema must be specified as a JSON string, containing an object that describes the fields expected at the root of each document. Each field can itself have more fields defined, allowing for nested structures: ```json { "Tag": "name=root, repetitiontype=REQUIRED", "Fields": [ {"Tag": "name=name, inname=NameIn, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag": "name=age, inname=Age, type=INT32, repetitiontype=REQUIRED"}, {"Tag": "name=id, inname=Id, type=INT64, repetitiontype=REQUIRED"}, {"Tag": "name=weight, inname=Weight, type=FLOAT, repetitiontype=REQUIRED"}, { "Tag": "name=favPokemon, inname=FavPokemon, type=LIST, repetitiontype=OPTIONAL", "Fields": [ {"Tag": "name=name, inname=PokeName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag": "name=coolness, inname=Coolness, type=FLOAT, repetitiontype=REQUIRED"} ] } ] } ``` A schema can be derived from a source file using [https://github.com/xitongsys/parquet-go/tree/master/tool/parquet-tools](https://github.com/xitongsys/parquet-go/tree/master/tool/parquet-tools): ```sh ./parquet-tools -cmd schema -file foo.parquet ``` ## [](#fields)Fields ### [](#compression)`compression` The type of compression to use when writing parquet files, this field is ignored when consuming parquet files. **Type**: `string` **Default**: `snappy` **Options**: `uncompressed`, `snappy`, `gzip`, `lz4`, `zstd` ### [](#operator)`operator` Determines whether the processor converts messages into a parquet file or expands parquet files into messages. Converting into JSON allows subsequent processors and mappings to convert the data into any other format. **Type**: `string` | Option | Summary | | --- | --- | | from_json | Compress a batch of JSON documents into a file. | | to_json | Expand a file into one or more JSON messages. | ### [](#schema)`schema` A schema used to describe the parquet files being generated or consumed, the format of the schema is a JSON document detailing the tag and fields of documents. The schema can be found at: [https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json](https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json). Either a `schema_file` or `schema` field must be specified when creating Parquet files via the `from_json` operator. **Type**: `string` ```yaml # Examples: schema: |- { "Tag": "name=root, repetitiontype=REQUIRED", "Fields": [ {"Tag":"name=name,inname=NameIn,type=BYTE_ARRAY,convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag":"name=age,inname=Age,type=INT32,repetitiontype=REQUIRED"} ] } ``` ### [](#schema_file)`schema_file` A file path containing a schema used to describe the parquet files being generated or consumed, the format of the schema is a JSON document detailing the tag and fields of documents. The schema can be found at: [https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json](https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json). Either a `schema_file` or `schema` field must be specified when creating Parquet files via the `from_json` operator. **Type**: `string` ```yaml # Examples: schema_file: schemas/foo.json ``` --- # Page 274: parse_log **URL**: https://docs.redpanda.com/connect/components/processors/parse_log.md --- # parse_log --- title: parse_log latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/parse_log page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parse_log.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parse_log.adoc description: Parses common log formats into structured data. This is easier and often much faster than grok. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Parses common log [Formats](#formats) into [structured data](#codecs). This is easier and often much faster than [`grok`](https://docs.redpanda.com/connect/components/processors/grok/). #### Common ```yml processor: label: "" parse_log: format: "" # No default (required) ``` #### Advanced ```yml processor: label: "" parse_log: format: "" # No default (required) best_effort: true allow_rfc3339: true default_year: current default_timezone: UTC ``` ## [](#fields)Fields ### [](#allow_rfc3339)`allow_rfc3339` Also accept timestamps in rfc3339 format while parsing. Applicable to format `syslog_rfc3164`. **Type**: `bool` **Default**: `true` ### [](#best_effort)`best_effort` Still returns partially parsed messages even if an error occurs. **Type**: `bool` **Default**: `true` ### [](#default_timezone)`default_timezone` Sets the strategy to decide the timezone for rfc3164 timestamps. Applicable to format `syslog_rfc3164`. This value should follow the [time.LoadLocation](https://golang.org/pkg/time/#LoadLocation) format. **Type**: `string` **Default**: `UTC` ### [](#default_year)`default_year` Sets the strategy used to set the year for rfc3164 timestamps. Applicable to format `syslog_rfc3164`. When set to `current` the current year will be set, when set to an integer that value will be used. Leave this field empty to not set a default year at all. **Type**: `string` **Default**: `current` ### [](#format)`format` A common log [format](#formats) to parse. **Type**: `string` **Options**: `syslog_rfc5424`, `syslog_rfc3164` ## [](#codecs)Codecs Currently the only supported structured data codec is `json`. ## [](#formats)Formats ### [](#syslog_rfc5424)`syslog_rfc5424` Attempts to parse a log following the [Syslog RFC5424](https://tools.ietf.org/html/rfc5424) spec. The resulting structured document may contain any of the following fields: - `message` (string) - `timestamp` (string, RFC3339) - `facility` (int) - `severity` (int) - `priority` (int) - `version` (int) - `hostname` (string) - `procid` (string) - `appname` (string) - `msgid` (string) - `structureddata` (object) ### [](#syslog_rfc3164)`syslog_rfc3164` Attempts to parse a log following the [Syslog rfc3164](https://tools.ietf.org/html/rfc3164) spec. The resulting structured document may contain any of the following fields: - `message` (string) - `timestamp` (string, RFC3339) - `facility` (int) - `severity` (int) - `priority` (int) - `hostname` (string) - `procid` (string) - `appname` (string) - `msgid` (string) --- # Page 275: processors **URL**: https://docs.redpanda.com/connect/components/processors/processors.md --- # processors --- title: processors latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/processors page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/processors.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/processors.adoc description: A processor grouping several sub-processors. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A processor grouping several sub-processors. Introduced in version 4.25.0. This processor is useful in situations where you want to collect several processors under a single resource identifier, whether it is for making your configuration easier to read and navigate, or for improving the testability of your configuration. The behavior of child processors will match exactly the behavior they would have under any other processors block. #### Common ```yml processor: label: "" processors: [] ``` #### Advanced ```yml processor: label: "" processors: [] ``` ## [](#examples)Examples ### [](#grouped-processing)Grouped Processing Imagine we have a collection of processors who cover a specific functionality. We could use this processor to group them together and make it easier to read and mock during testing by giving the whole block a label: ```yaml pipeline: processors: - label: my_super_feature processors: - log: message: "Let's do something cool" - archive: format: json_array - mapping: root.items = this ``` --- # Page 276: protobuf **URL**: https://docs.redpanda.com/connect/components/processors/protobuf.md --- # protobuf --- title: protobuf latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/protobuf page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/protobuf.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/protobuf.adoc description: Performs conversions to or from a protobuf message. This processor uses reflection, meaning conversions can be made directly from the target .proto files. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Handles conversions between JSON documents and protobuf messages using reflection, which allows you to make conversions from or to the target `.proto` files. For more information about JSON mapping of protobuf messages, see [ProtoJSON Format](https://protobuf.dev/programming-guides/json/) and [Examples](#examples). #### Common ```yml processor: label: "" protobuf: operator: "" # No default (required) message: "" # No default (required) discard_unknown: false use_proto_names: false import_paths: [] use_enum_numbers: false bsr: [] ``` #### Advanced ```yml processor: label: "" protobuf: operator: "" # No default (required) message: "" # No default (required) discard_unknown: false use_proto_names: false import_paths: [] use_enum_numbers: false bsr: [] ``` ## [](#performance-considerations)Performance considerations Processing protobuf messages using reflection is less performant than using generated native code. For scenarios where performance is critical, consider using [Redpanda Connect plugins](https://github.com/benthosdev/benthos-plugin-example). ## [](#operators)Operators ### [](#to_json)`to_json` Converts protobuf messages into a generic JSON structure, which makes it easier to manipulate the contents of the JSON document within Redpanda Connect. ### [](#from_json)`from_json` Attempts to create a target protobuf message from a generic JSON structure. ## [](#fields)Fields ### [](#bsr)`bsr[]` Buf Schema Registry configuration. Either this field or `import_paths` must be populated. Note that this field is an array, and multiple BSR configurations can be provided. Requires version 4.63.0 or later. **Type**: `array` **Default**: `[]` ### [](#bsr-api_key)`bsr[].api_key` Buf Schema Registry server API key, can be left blank for a public registry. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#bsr-module)`bsr[].module` Module to fetch from a Buf Schema Registry, for example, 'buf.build/exampleco/mymodule'. **Type**: `string` ### [](#bsr-url)`bsr[].url` Buf Schema Registry URL, leave blank to extract from module. **Type**: `string` **Default**: `""` ### [](#bsr-version)`bsr[].version` Version to retrieve from the Buf Schema Registry, leave blank for latest. **Type**: `string` **Default**: `""` ### [](#discard_unknown)`discard_unknown` When set to `true`, the `from_json` operator discards fields that are unknown to the schema. Requires version 4.24.0 or later. **Type**: `bool` **Default**: `false` ### [](#import_paths)`import_paths[]` A list of directories that contain `.proto` files, including all definitions required for parsing the target message. If left empty, the current directory is used. This processor walks each listed directory and imports all `.proto` files it finds. Either this field or `bsr` must be populated. **Type**: `array` **Default**: `[]` ### [](#message)`message` The fully-qualified name of the protobuf message to convert from or to JSON. **Type**: `string` ### [](#operator)`operator` The operator to execute. **Type**: `string` **Options**: `to_json`, `from_json`, `decode` ### [](#use_enum_numbers)`use_enum_numbers` When set to `true`, the `to_json` or `decode` operator deserializes enumeration fields as their numerical values instead of their string names. For example, an enum field with a value of `ENUM_VALUE_ONE` is represented as `1` in the JSON output. Requires version 4.53.0 or later. **Type**: `bool` **Default**: `false` ### [](#use_proto_names)`use_proto_names` When set to `true`, the `to_json` or `decode` operator deserializes fields exactly as named in the schema file. Requires version 4.24.0 or later. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#json-to-protobuf-using-schema-from-disk)JSON to Protobuf using Schema from Disk If we have the following protobuf definition within a directory called `testing/schema`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of JSON documents of the form: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` We can convert the documents into protobuf messages with the following config: ```yaml pipeline: processors: - protobuf: operator: from_json message: testing.Person import_paths: [ testing/schema ] ``` ### [](#protobuf-to-json-using-schema-from-disk)Protobuf to JSON using Schema from Disk If we have the following protobuf definition within a directory called `testing/schema`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of protobuf messages of the type `Person`, we could convert them into JSON documents of the format: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` With the following config: ```yaml pipeline: processors: - protobuf: operator: to_json message: testing.Person import_paths: [ testing/schema ] ``` ### [](#json-to-protobuf-using-buf-schema-registry)JSON to Protobuf using Buf Schema Registry If we have the following protobuf definition within a BSR module hosted at `buf.build/exampleco/mymodule`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of JSON documents of the form: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` We can convert the documents into protobuf messages with the following config: ```yaml pipeline: processors: - protobuf: operator: from_json message: testing.Person bsr: - module: buf.build/exampleco/mymodule api_key: xxx ``` ### [](#protobuf-to-json-using-buf-schema-registry)Protobuf to JSON using Buf Schema Registry If we have the following protobuf definition within a BSR module hosted at `buf.build/exampleco/mymodule`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of protobuf messages of the type `Person`, we could convert them into JSON documents of the format: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` With the following config: ```yaml pipeline: processors: - protobuf: operator: to_json message: testing.Person bsr: - module: buf.build/exampleco/mymodule api_key: xxxx ``` --- # Page 277: qdrant **URL**: https://docs.redpanda.com/connect/components/processors/qdrant.md --- # qdrant --- title: qdrant latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/qdrant page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/qdrant.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/qdrant.adoc description: Query items within a Qdrant collection. page-git-created-date: "2025-05-19" page-git-modified-date: "2026-10-10" --- Query items within a [Qdrant](https://qdrant.tech/) collection. Introduced in version 4.54.0. #### Common ```yml processor: label: "" qdrant: grpc_host: "" # No default (required) api_token: "" collection_name: "" # No default (required) vector_mapping: "" # No default (required) filter: "" # No default (optional) payload_fields: [] payload_filter: include limit: 10 ``` #### Advanced ```yml processor: label: "" qdrant: grpc_host: "" # No default (required) api_token: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] collection_name: "" # No default (required) vector_mapping: "" # No default (required) filter: "" # No default (optional) payload_fields: [] payload_filter: include limit: 10 ``` ## [](#fields)Fields ### [](#api_token)`api_token` The Qdrant API token to use for authentication, which defaults to an empty string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#collection_name)`collection_name` The name of the Qdrant collection you want to query. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#filter)`filter` Specify additional filtering to perform on returned results. Mappings must return a valid Qdrant filter using the proto3-encoded form. For examples, see the [Qdrant documentation](https://qdrant.tech/documentation/concepts/filtering/). **Type**: `string` ```yaml # Examples: filter: | root.must = [ {"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}}, {"field":{"key": "city", "match": {"text": "London"}}}, ] # --- filter: | root.must = [ {"field":{"key": "city", "match": {"text": "London"}}}, ] root.must_not = [ {"field":{"color": "city", "match": {"text": "red"}}}, ] ``` ### [](#grpc_host)`grpc_host` The gRPC host of the Qdrant server. **Type**: `string` ```yaml # Examples: grpc_host: localhost:6334 # --- grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334 ``` ### [](#limit)`limit` The maximum number of points to return from the collection. **Type**: `int` **Default**: `10` ### [](#payload_fields)`payload_fields[]` The fields to include or exclude in returned results. Use this field in combination with `payload_filter`. **Type**: `array` **Default**: `[]` ### [](#payload_filter)`payload_filter` Whether to include or exclude the fields specified in `payload_fields` from the returned results. **Type**: `string` **Default**: `include` | Option | Summary | | --- | --- | | exclude | Exclude the payload fields specified in payload_fields. | | include | Include the payload fields specified in payload_fields. | ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#vector_mapping)`vector_mapping` A mapping that extracts the search vector from the input message. The processor uses this vector to query the collection. **Type**: `string` ```yaml # Examples: vector_mapping: root = [1.2, 0.5, 0.76] # --- vector_mapping: root = this.vector # --- vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]] # --- vector_mapping: 'root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}}' # --- vector_mapping: 'root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]}' # --- vector_mapping: 'root = {"some_dense": [0.352,0.532,0.532,0.234]}' ``` --- # Page 278: rate_limit **URL**: https://docs.redpanda.com/connect/components/processors/rate_limit.md --- # rate_limit --- title: rate_limit latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/rate_limit page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/rate_limit.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/rate_limit.adoc description: Throttles the throughput of a pipeline according to a specified rate_limit resource. Rate limits are shared across components and therefore apply globally to all processing pipelines. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Throttles the throughput of a pipeline according to a specified [`rate_limit`](https://docs.redpanda.com/connect/components/rate_limits/about/) resource. Rate limits are shared across components and therefore apply globally to all processing pipelines. #### Common ```yml processor: label: "" rate_limit: resource: "" # No default (required) ``` #### Advanced ```yml processor: label: "" rate_limit: resource: "" # No default (required) ``` ## [](#fields)Fields ### [](#resource)`resource` The target [`rate_limit` resource](https://docs.redpanda.com/connect/components/rate_limits/about/). **Type**: `string` --- # Page 279: redis_script **URL**: https://docs.redpanda.com/connect/components/processors/redis_script.md --- # redis_script --- title: redis_script latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/redis_script page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/redis_script.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/redis_script.adoc description: Performs actions against Redis using LUA scripts. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs actions against Redis using [LUA scripts](https://redis.io/docs/latest/develop/programmability/eval-intro/). Introduced in version 4.11.0. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/). #### Common ```yml processor: label: "" redis_script: url: "" # No default (required) script: "" # No default (required) args_mapping: "" # No default (required) keys_mapping: "" # No default (required) ``` #### Advanced ```yml processor: label: "" redis_script: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] script: "" # No default (required) args_mapping: "" # No default (required) keys_mapping: "" # No default (required) retries: 3 retry_period: 500ms ``` ## [](#examples)Examples ### [](#running-a-script)Running a script The following example will use a script execution to get next element from a sorted set and set its score with timestamp unix nano value. ```yaml pipeline: processors: - redis_script: url: TODO script: | local value = redis.call("ZRANGE", KEYS[1], '0', '0') if next(elements) == nil then return '' end redis.call("ZADD", "XX", KEYS[1], ARGV[1], value) return value keys_mapping: 'root = [ meta("key") ]' args_mapping: 'root = [ timestamp_unix_nano() ]' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that evaluates to an array of values passed to the script, where they are available as `ARGV[1]`, `ARGV[2]`, and so on. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.key ] # --- args_mapping: root = [ meta("kafka_key"), "hardcoded_value" ] ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#keys_mapping)`keys_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to an array of keys matching in size to the number of arguments required for the specified Redis script. **Type**: `string` ```yaml # Examples: keys_mapping: root = [ this.key ] # --- keys_mapping: root = [ meta("kafka_key"), this.count ] ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#retries)`retries` The maximum number of retries before abandoning a request. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The time to wait before consecutive retry attempts. **Type**: `string` **Default**: `500ms` ### [](#script)`script` A script to use for the target operator. It has precedence over the 'command' field. **Type**: `string` ```yaml # Examples: script: return redis.call('set', KEYS[1], ARGV[1]) ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 280: redis **URL**: https://docs.redpanda.com/connect/components/processors/redis.md --- # redis --- title: redis latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/redis page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/redis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/redis.adoc description: Performs actions against Redis that aren't possible using a cache processor. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a branch processor. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Performs actions against Redis that aren’t possible using a [`cache`](https://docs.redpanda.com/connect/components/processors/cache/) processor. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/). #### Common ```yml processor: label: "" redis: url: "" # No default (required) command: "" # No default (optional) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] command: "" # No default (optional) args_mapping: "" # No default (optional) retries: 3 retry_period: 500ms ``` ## [](#examples)Examples ### [](#querying-cardinality)Querying Cardinality If given payloads containing a metadata field `set_key` it’s possible to query and store the cardinality of the set for each message using a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) in order to augment rather than replace the message contents: ```yaml pipeline: processors: - branch: processors: - redis: url: TODO command: scard args_mapping: 'root = [ meta("set_key") ]' result_map: 'root.cardinality = this' ``` ### [](#running-total)Running Total If we have JSON data containing number of friends visited during covid 19: ```json {"name":"ash","month":"feb","year":2019,"friends_visited":10} {"name":"ash","month":"apr","year":2019,"friends_visited":-2} {"name":"bob","month":"feb","year":2019,"friends_visited":3} {"name":"bob","month":"apr","year":2019,"friends_visited":1} ``` We can add a field that contains the running total number of friends visited: ```json {"name":"ash","month":"feb","year":2019,"friends_visited":10,"total":10} {"name":"ash","month":"apr","year":2019,"friends_visited":-2,"total":8} {"name":"bob","month":"feb","year":2019,"friends_visited":3,"total":3} {"name":"bob","month":"apr","year":2019,"friends_visited":1,"total":4} ``` Using the `incrby` command: ```yaml pipeline: processors: - branch: processors: - redis: url: TODO command: incrby args_mapping: 'root = [ this.name, this.friends_visited ]' result_map: 'root.total = this' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that evaluates to an array of arguments for `command`, such as the key followed by any values. The array must contain as many values as the Redis command requires. Requires version 4.3.0 or later. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.key ] # --- args_mapping: root = [ meta("kafka_key"), this.count ] ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command to execute. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). Requires version 4.3.0 or later. **Type**: `string` ```yaml # Examples: command: scard # --- command: incrby # --- command: ${! meta("command") } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#retries)`retries` The maximum number of retries before abandoning a request. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The time to wait before consecutive retry attempts. **Type**: `string` **Default**: `500ms` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 281: redpanda_data_transform **URL**: https://docs.redpanda.com/connect/components/processors/redpanda_data_transform.md --- # redpanda_data_transform --- title: redpanda_data_transform latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/redpanda_data_transform page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/redpanda_data_transform.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/redpanda_data_transform.adoc description: Executes a Redpanda Data Transform as a processor. page-git-created-date: "2024-08-15" page-git-modified-date: "2026-10-10" --- Executes a Redpanda Data Transform as a processor Introduced in version 4.31.0. #### Common ```yml processor: label: "" redpanda_data_transform: module_path: "" # No default (required) input_key: "" # No default (optional) output_key: "" # No default (optional) input_headers: include_prefixes: [] include_patterns: [] output_metadata: include_prefixes: [] include_patterns: [] ``` #### Advanced ```yml processor: label: "" redpanda_data_transform: module_path: "" # No default (required) input_key: "" # No default (optional) output_key: "" # No default (optional) input_headers: include_prefixes: [] include_patterns: [] output_metadata: include_prefixes: [] include_patterns: [] timestamp: "" # No default (optional) timeout: 10s max_memory_pages: 1600 ``` This processor executes a Redpanda Data Transform WebAssembly module, calling OnRecordWritten for each message being processed. You can find out about how transforms work here: [](../../../../streaming/current/develop/data-transforms/how-transforms-work/)[https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/](https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/) ## [](#fields)Fields ### [](#input_headers)`input_headers` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#input_headers-include_patterns)`input_headers.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#input_headers-include_prefixes)`input_headers.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#input_key)`input_key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_memory_pages)`max_memory_pages` The maximum amount of wasm memory pages (64KiB) that an individual wasm module instance can use **Type**: `int` **Default**: `1600` ### [](#module_path)`module_path` The path of the compiled Redpanda Data Transform module to run, such as the `.wasm` file built by `rpk transform build`. **Type**: `string` ### [](#output_key)`output_key` An optional name of metadata for an output message key. **Type**: `string` ### [](#output_metadata)`output_metadata` Determine which (if any) message headers should be added to the output as metadata. **Type**: `object` ### [](#output_metadata-include_patterns)`output_metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#output_metadata-include_prefixes)`output_metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#timeout)`timeout` The maximum period of time for a message to be processed **Type**: `string` **Default**: `10s` ### [](#timestamp)`timestamp` An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp: ${! timestamp_unix_milli() } # --- timestamp: ${! metadata("kafka_timestamp_ms") } ``` --- # Page 282: resource **URL**: https://docs.redpanda.com/connect/components/processors/resource.md --- # resource --- title: resource latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/resource page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/resource.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/resource.adoc description: Resource is a processor type that runs a processor resource identified by its label. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Resource is a processor type that runs a processor resource identified by its label. ## Common ```yml processor: resource: "" ``` ## Advanced ```yml processor: resource: "" ``` This processor allows you to reference the same configured processor resource in multiple places, and can also tidy up large nested configs. For example, the config: ```yaml pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() ``` Is equivalent to: ```yaml pipeline: processors: - resource: foo_proc processor_resources: - label: foo_proc mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() ``` You can find out more about resources in [Resources](https://docs.redpanda.com/connect/configuration/resources/) --- # Page 283: retry **URL**: https://docs.redpanda.com/connect/components/processors/retry.md --- # retry --- title: retry latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/retry page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/retry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/retry.adoc description: Attempts to execute a series of child processors until success. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > This component is in beta. Its behavior might change in a future release. Attempts to execute a series of child processors until success. Introduced in version 4.27.0. Executes child processors and if a resulting message is errored then, after a specified backoff period, the same original message will be attempted again through those same processors. If the child processors result in more than one message then the retry mechanism will kick in if _any_ of the resulting messages are errored. It is important to note that any mutations performed on the message during these child processors will be discarded for the next retry, and therefore it is safe to assume that each execution of the child processors will always be performed on the data as it was when it first reached the retry processor. By default the retry backoff has a specified [`max_elapsed_time`](#backoffmax_elapsed_time), if this time period is reached during retries and an error still occurs these errored messages will proceed through to the next processor after the retry (or your outputs). Normal [error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/) can be used on these messages. In order to avoid permanent loops any error associated with messages as they first enter a retry processor will be cleared. ## [](#metadata)Metadata This processor adds the following metadata fields to each message: - `retry_count` - The number of retry attempts. - `backoff_duration` - The total time (in nanoseconds) elapsed while performing retries. > ⚠️ **CAUTION: Batching** > > Batching > > If you wish to wrap a batch-aware series of processors then take a look at the [batching section](#batching). ### Common ```yml processor: label: "" retry: backoff: initial_interval: 500ms max_interval: 10s max_elapsed_time: 1m processors: [] # No default (required) parallel: false max_retries: 0 ``` ### Advanced ```yml processor: label: "" retry: backoff: initial_interval: 500ms max_interval: 10s max_elapsed_time: 1m processors: [] # No default (required) parallel: false max_retries: 0 ``` ## [](#examples)Examples ### [](#stop-ignoring-me-taz)Stop ignoring me Taz Here we have a config where I generate animal noises and send them to Taz via HTTP. Taz has a tendency to stop his servers whenever I dispatch my animals upon him, and therefore these HTTP requests sometimes fail. However, I have the retry processor and with this super power I can specify a back off policy and it will ensure that for each animal noise the HTTP processor is attempted until either it succeeds or my Redpanda Connect instance is stopped. I even go as far as to zero-out the maximum elapsed time field, which means that for each animal noise I will wait indefinitely, because I really really want Taz to receive every single animal noise that he is entitled to. ```yaml input: generate: interval: 1s mapping: 'root.noise = [ "woof", "meow", "moo", "quack" ].index(random_int(min: 0, max: 3))' pipeline: processors: - retry: backoff: initial_interval: 100ms max_interval: 5s max_elapsed_time: 0s processors: - http: url: 'http://example.com/try/not/to/dox/taz' verb: POST output: # Drop everything because it's junk data, I don't want it lol drop: {} ``` ## [](#fields)Fields ### [](#backoff)`backoff` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. Setting this value to a zeroed duration (such as `0s`) will result in unbounded retries. **Type**: `string` **Default**: `1m` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `10s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#max_retries)`max_retries` The maximum number of retry attempts before the request is aborted. Setting this value to `0` will result in unbounded number of retries. Requires version 4.31.0 or later. **Type**: `int` **Default**: `0` ### [](#parallel)`parallel` When processing batches of messages these batches are ignored and the processors apply to each message sequentially. However, when this field is set to `true` each message will be processed in parallel. Caution should be made to ensure that batch sizes do not surpass a point where this would cause resource (CPU, memory, API limits) contention. **Type**: `bool` **Default**: `false` ### [](#processors)`processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to execute on each message. **Type**: `array` ## [](#batching)Batching When messages are batched the child processors of a retry are executed for each individual message in isolation, performed serially by default but in parallel when the field [`parallel`](#parallel) is set to `true`. This is an intentional limitation of the retry processor and is done in order to ensure that errors are correctly associated with a given input message. Otherwise, the archiving, expansion, grouping, filtering and so on of the child processors could obfuscate this relationship. If the target behavior of your retried processors is "batch aware", in that you wish to perform some processing across the entire batch of messages and repeat it in the event of errors, you can use an [`archive` processor](https://docs.redpanda.com/connect/components/processors/archive/) to collapse the batch into an individual message. Then, within these child processors either perform your batch aware processing on the archive, or use an [`unarchive` processor](https://docs.redpanda.com/connect/components/processors/unarchive/) in order to expand the single message back out into a batch. For example, if the retry processor were being used to wrap an HTTP request where the payload data is a batch archived into a JSON array it should look something like this: ```yaml pipeline: processors: - archive: format: json_array - retry: processors: - http: url: example.com/nope verb: POST - unarchive: format: json_array ``` --- # Page 284: schema_registry_decode **URL**: https://docs.redpanda.com/connect/components/processors/schema_registry_decode.md --- # schema_registry_decode --- title: schema_registry_decode latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/schema_registry_decode page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/schema_registry_decode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/schema_registry_decode.adoc description: Automatically decodes and validates messages with schemas from a Confluent Schema Registry service. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Automatically decodes and validates messages with schemas from a Confluent Schema Registry service. This processor uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). #### Common ```yml processor: label: "" schema_registry_decode: avro: raw_unions: false # No default (optional) preserve_logical_types: false translate_kafka_connect_types: false store_schema_metadata: "" # No default (optional) protobuf: use_proto_names: false use_enum_numbers: false emit_unpopulated: false emit_default_values: false serialize_to_json: true json: coerce_data: false cache_duration: 10m url: "" # No default (required) default_schema_id: 0 # No default (optional) ``` #### Advanced ```yml processor: label: "" schema_registry_decode: avro: raw_unions: false # No default (optional) preserve_logical_types: false translate_kafka_connect_types: false mapping: "" # No default (optional) store_schema_metadata: "" # No default (optional) protobuf: use_proto_names: false use_enum_numbers: false emit_unpopulated: false emit_default_values: false serialize_to_json: true json: coerce_data: false cache_duration: 10m url: "" # No default (required) default_schema_id: 0 # No default (optional) oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` Decodes messages automatically from a schema stored within a [Confluent Schema Registry service](https://docs.confluent.io/platform/current/schema-registry/index.html) by extracting a schema ID from the message and obtaining the associated schema from the registry. If a message fails to match against the schema then it will remain unchanged and the error can be caught using [error-handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). Avro, Protobuf and JSON schemas are supported, all are capable of expanding from schema references as of v4.22.0. ## [](#avro-json-format)Avro JSON format By default, this processor expects documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when decoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The name is the type’s name, and the value is the recursively-encoded value. The user-specified name is used for Avro’s named types (record, fixed, or enum). For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can create documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`avro.raw_unions`](#avro-raw_unions) to `true`. ## [](#protobuf-format)Protobuf format This processor decodes Protobuf messages to JSON documents. For more information about the JSON mapping of Protobuf messages, see the [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3#json). ## [](#metadata)Metadata This processor also adds the following metadata to each outgoing message: schema\_id: the ID of the schema in the schema registry that was associated with the message. ## [](#fields)Fields ### [](#avro)`avro` Configuration for how to decode schemas that are of type AVRO. Requires version 4.46.0 or later. **Type**: `object` ### [](#avro-mapping)`avro.mapping` Define a custom mapping to apply to the JSON representation of Avro schemas. You can use mappings to convert custom types emitted by other tools, such as Debezium, into standard Avro types. Requires version 4.47.0 or later. **Type**: `string` ```yaml # Examples: mapping: | map isDebeziumTimestampType { root = this.type == "long" && this."connect.name" == "io.debezium.time.Timestamp" && !this.exists("logicalType") } map debeziumTimestampToAvroTimestamp { let mapped_fields = this.fields.or([]).map_each(item -> item.apply("debeziumTimestampToAvroTimestamp")) root = match { this.type == "record" => this.assign({"fields": $mapped_fields}) this.type.type() == "array" => this.assign({"type": this.type.map_each(item -> item.apply("debeziumTimestampToAvroTimestamp"))}) # Add a logical type so that it's decoded as a timestamp instead of a long. this.type.type() == "object" && this.type.apply("isDebeziumTimestampType") => this.merge({"type":{"logicalType": "timestamp-millis"}}) _ => this } } root = this.apply("debeziumTimestampToAvroTimestamp") ``` ### [](#avro-preserve_logical_types)`avro.preserve_logical_types` Choose whether to: - Transform logical types into their primitive type (default). For example, decimals become raw bytes and timestamps become plain integers. - Preserve logical types. Set to `true` to preserve logical types, which keeps decimal types as numbers in Bloblang and timestamps as time values. **Type**: `bool` **Default**: `false` ### [](#avro-raw_unions)`avro.raw_unions` Whether Avro messages should be decoded into normal JSON (JSON that meets the expectations of regular internet JSON) rather than [Avro JSON](https://avro.apache.org/docs/current/specification/#json-encoding). If set to `false` (default), Avro messages are decoded as Avro JSON. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would be decoded as: - A `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance. If set to `true`, Avro messages are decoded as standard JSON. For example, the same union schema `["null","string","Transaction"]` is decoded as: - A `null` as JSON `null` - The string `"a"` as `"a"` - A `Transaction` instance as `{…​}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance. **Type**: `bool` ### [](#avro-store_schema_metadata)`avro.store_schema_metadata` Optionally store the schema used to decode messages as a metadata field under the given name. This field can later be referenced in other components such as a `parquet_encode` processor in order to automatically infer their schema. Requires version 4.62.0 or later. **Type**: `string` ### [](#avro-translate_kafka_connect_types)`avro.translate_kafka_connect_types` Only valid if preserve\_logical\_types is true. This decodes various Kafka Connect types into their bloblang equivalents when not representable by standard logical types according to the Avro standard. Types that are currently translated: | Type Name | Bloblang Type | Description | | --- | --- | --- | | io.debezium.time.Date | timestamp | Date without time (days since epoch) | | io.debezium.time.Timestamp | timestamp | Timestamp without timezone (milliseconds since epoch) | | io.debezium.time.MicroTimestamp | timestamp | Timestamp with microsecond precision | | io.debezium.time.NanoTimestamp | timestamp | Timestamp with nanosecond precision | | io.debezium.time.ZonedTimestamp | timestamp | Timestamp with timezone (ISO-8601 format) | | io.debezium.time.Year | timestamp at January 1st at 00:00:00 | Year value | | io.debezium.time.Time | timestamp at the unix epoch | Time without date (milliseconds past midnight) | | io.debezium.time.MicroTime | timestamp at the unix epoch | Time with microsecond precision | | io.debezium.time.NanoTime | timestamp at the unix epoch | Time with nanosecond precision | Requires version 4.55.0 or later. **Type**: `bool` **Default**: `false` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. Requires version 4.7.0 or later. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#cache_duration)`cache_duration` The duration after which a cached schema is considered stale and is removed from the cache. Requires version 4.54.0 or later. **Type**: `string` **Default**: `10m` ```yaml # Examples: cache_duration: 1h # --- cache_duration: 5m ``` ### [](#default_schema_id)`default_schema_id` This schema ID is used when a message’s schema header cannot be read (`ErrBadHeader`). If this value is not set, schema header errors are returned. This configuration does not work with protobuf schemas. > 💡 **TIP** > > You can also use the [`with_schema_registry_header`](https://docs.redpanda.com/connect/guides/bloblang/functions/#with_schema_registry_header) bloblang function to add a schema ID to messages. Requires version 4.64.0 or later. **Type**: `int` ### [](#json)`json` Configuration for how to decode schemas that are of type JSON. Requires version 4.97.0 or later. **Type**: `object` ### [](#json-coerce_data)`json.coerce_data` Whether decoded values should be coerced to match the types declared in the JSON Schema. By default JSON Schema decoding only validates the message and leaves it untouched, which means numbers are later interpreted as floating point (`double`) and date-time values as strings. When set to `true` the decoder rebuilds the message so that values match the schema: `integer` fields become 64-bit integers, `number` fields stay floating point, `string` fields with `format: date-time` become timestamps, and any `default` values declared in the schema are applied to absent fields. This is useful for downstream components that infer their schema from the decoded values, such as the `iceberg` outputs, which will then create `bigint` columns for integer fields rather than `double`. Note that, unlike the default behavior, this is no longer a read-only operation: the message contents are transformed. Because coercion is stricter than validation, a message that passes validation may still fail coercion (for example an integer that overflows a 64-bit value, or a `date-time` string that is not valid RFC 3339), in which case the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). **Type**: `bool` **Default**: `false` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. Requires version 4.7.0 or later. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. Requires version 4.7.0 or later. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#protobuf)`protobuf` Configuration for how to decode schemas that are of type PROTOBUF. Requires version 4.57.0 or later. **Type**: `object` ### [](#protobuf-emit_default_values)`protobuf.emit_default_values` Whether to emit default-valued primitive fields, empty lists, and empty maps. emit\_unpopulated takes precedence over emit\_default\_values **Type**: `bool` **Default**: `false` ### [](#protobuf-emit_unpopulated)`protobuf.emit_unpopulated` Whether to emit unpopulated fields. It does not emit unpopulated oneof fields or unpopulated extension fields. **Type**: `bool` **Default**: `false` ### [](#protobuf-serialize_to_json)`protobuf.serialize_to_json` If messages should be serialized to JSON bytes. If false then the message is kept in decoded form, which means that 64 bit integers are not converted to strings and types for bytes and google.protobuf.Timestamp are preserved (as they are not serialized to JSON strings). Requires version 4.69.0 or later. **Type**: `bool` **Default**: `true` ### [](#protobuf-use_enum_numbers)`protobuf.use_enum_numbers` Emits enum values as numbers. **Type**: `bool` **Default**: `false` ### [](#protobuf-use_proto_names)`protobuf.use_proto_names` Use proto field name instead of lowerCamelCase name. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 285: schema_registry_encode **URL**: https://docs.redpanda.com/connect/components/processors/schema_registry_encode.md --- # schema_registry_encode --- title: schema_registry_encode latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/schema_registry_encode page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/schema_registry_encode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/schema_registry_encode.adoc description: Automatically encodes and validates messages with schemas from a Confluent Schema Registry service. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Automatically encodes and validates messages with schemas from a Confluent Schema Registry service. This processor uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). Introduced in version 3.58.0. #### Common ```yml processor: label: "" schema_registry_encode: url: "" # No default (required) subject: "" # No default (required) refresh_period: 10m schema_metadata: "" format: "" # No default (optional) avro: raw_json: false # No default (optional) record_name: "" namespace: "" ``` #### Advanced ```yml processor: label: "" schema_registry_encode: url: "" # No default (required) subject: "" # No default (required) refresh_period: 10m schema_metadata: "" format: "" # No default (optional) normalize: true avro: raw_json: false # No default (optional) input_encoding: auto record_name: "" namespace: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` Encodes messages automatically from schemas obtains from a [Confluent Schema Registry service](https://docs.confluent.io/platform/current/schema-registry/index.html) by polling the service for the latest schema version for target subjects. If a message fails to encode under the schema then it will remain unchanged and the error can be caught using [error-handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). Avro, Protobuf and JSON schemas are supported, all are capable of expanding from schema references as of v4.22.0. ## [](#avro-json-format)Avro JSON format By default, this processor expects documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when encoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The name is the type’s name, and the value is the recursively-encoded value. The user-specified name is used for Avro’s named types (record, fixed, or enum). For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - A `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can consume documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`avro_raw_json`](#avro-raw_json) to `true`. ### [](#known-issues)Known issues Important! There is an outstanding issue in the [avro serializing library](https://github.com/linkedin/goavro) that Redpanda Connect uses which means it [doesn’t encode logical types correctly](https://github.com/linkedin/goavro/issues/252). It’s still possible to encode logical types that are in-line with the spec if `avro_raw_json` is set to true, though now of course non-logical types will not be in-line with the spec. ## [](#protobuf-format)Protobuf format This processor encodes Protobuf messages either from any format parsed within Redpanda Connect (encoded as JSON by default), or from raw JSON documents. For more information about the JSON mapping of Protobuf messages, see the [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3#json). ### [](#multiple-message-support)Multiple message support When a target subject presents a Protobuf schema that contains multiple messages it becomes ambiguous which message definition a given input data should be encoded against. In such scenarios Redpanda Connect will attempt to encode the data against each of them and select the first to successfully match against the data, this process currently **ignores all nested message definitions**. In order to speed up this exhaustive search the last known successful message will be attempted first for each subsequent input. We will be considering alternative approaches in future so please [get in touch](https://redpanda.com/slack) with thoughts and feedback. ## [](#fields)Fields ### [](#avro)`avro` Configuration for Avro encoding. Requires version 4.83.0 or later. **Type**: `object` ### [](#avro-input_encoding)`avro.input_encoding` How the values in each message should be read. Avro JSON and plain JSON disagree about strings, and neither reading is a superset of the other. Avro JSON spells a `bytes` or `fixed` value as one codepoint per byte, so the unscaled decimal `0x21` is the string `"!"`; the plain reading takes a string for a decimal as decimal notation, so `"3.33"` means 3.33. The same JSON string is therefore a different value under each, and only the pipeline knows which was meant. - `auto` (the default) picks the reader from the form each message arrives in: a raw payload is read as Avro JSON, a structured message as Go values. This is right for `schema_registry_decode` feeding this processor directly, and for CDC sources, but a processor in between (a `mapping`, a `branch`) parses the payload and leaves the message structured, at which point its Avro JSON values are read the plain way and a bytes-backed decimal fails to encode. - `avro_json` reads every message as Avro JSON. Set this when the data came from `schema_registry_decode`, including when something between the two processors has parsed it. A message that carries Go values Avro JSON has no spelling for (a `[]byte` from `content()` or `decode("base64")`, or the `[]byte` and `time.Time` values that `preserve_logical_types` produces for bytes, fixed and timestamp fields) is rejected rather than encoded as the wrong value, so a pipeline that mixes those with Avro JSON values has no mode that reads both: map such fields to their Avro JSON spelling first, or split the encode. - `native` reads every message as Go and plain JSON values. Set this for hand-written JSON and for sources that spell decimals in decimal notation. This is independent of `raw_json`, which does not affect how values are read. Requires version 4.108.0 or later. **Type**: `string` **Default**: `auto` **Options**: `auto`, `avro_json`, `native` ### [](#avro-namespace)`avro.namespace` The Avro namespace for the root record type when encoding from a common schema (schema\_metadata mode). **Type**: `string` **Default**: `""` ### [](#avro-raw_json)`avro.raw_json` Whether messages encoded in Avro format should be parsed as normal JSON rather than Avro JSON. Overrides the deprecated top-level `avro_raw_json` when set. **Type**: `bool` ### [](#avro-record_name)`avro.record_name` The name to use for the root Avro record type when encoding from a common schema (schema\_metadata mode). If empty, derived from the subject. **Type**: `string` **Default**: `""` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component. Requires version 4.7.0 or later. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#format)`format` The encoding format to use when converting a common schema from metadata. Required when `schema_metadata` is set. Requires version 4.83.0 or later. **Type**: `string` **Options**: `avro`, `json_schema` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. Requires version 4.7.0 or later. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#normalize)`normalize` Whether to normalize the schema before registering with the schema registry (schema\_metadata mode only). Requires version 4.83.0 or later. **Type**: `bool` **Default**: `true` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. Requires version 4.7.0 or later. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#refresh_period)`refresh_period` The period after which a schema is refreshed for each subject, this is done by polling the schema registry service. **Type**: `string` **Default**: `10m` ```yaml # Examples: refresh_period: 60s # --- refresh_period: 1h ``` ### [](#schema_metadata)`schema_metadata` When set, the processor reads a schema in benthos common schema format from this metadata key on each message, converts it to the format specified by `format`, registers it with the schema registry under the configured subject, and encodes the message. When empty (the default), the processor pulls the latest schema from the registry instead. Requires version 4.83.0 or later. **Type**: `string` **Default**: `""` ### [](#subject)`subject` The schema subject to derive schemas from. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo # --- subject: ${! meta("kafka_topic") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 286: select_parts **URL**: https://docs.redpanda.com/connect/components/processors/select_parts.md --- # select_parts --- title: select_parts latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/select_parts page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/select_parts.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/select_parts.adoc description: Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored. The selected parts are added to the new message batch in the same order as the selection array. E.g. with 'parts' set to \[ 2, 0, 1 \] and the message parts \[ '0', '1', '2', '3' \], the output will be \[ '2', '0', '1' \]. If none of the selected parts exist in the input batch (resulting in an empty output message) the batch is dropped entirely. Message indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if index = -1 then the selected part will be the last part of the message, if index = -2 then the part before the last element with be selected, and so on. This processor is only applicable to [batched messages](https://docs.redpanda.com/connect/configuration/batching/). #### Common ```yml processor: label: "" select_parts: parts: [] ``` #### Advanced ```yml processor: label: "" select_parts: parts: [] ``` ## [](#fields)Fields ### [](#parts)`parts[]` An array of message indexes of a batch. Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. **Type**: `array` **Default**: `[]` --- # Page 287: sentry_capture **URL**: https://docs.redpanda.com/connect/components/processors/sentry_capture.md --- # sentry_capture --- title: sentry_capture latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sentry_capture page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sentry_capture.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sentry_capture.adoc description: Captures log events from messages and submits them to Sentry. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Captures log events from messages and submits them to [Sentry](https://sentry.io/). Introduced in version 4.16.0. #### Common ```yml processor: label: "" sentry_capture: dsn: "" message: "" # No default (required) context: "" # No default (optional) extras: "" # No default (optional) tags: {} # No default (optional) environment: "" release: "" level: INFO transport_mode: async flush_timeout: 5s sampling_rate: 1 ``` #### Advanced ```yml processor: label: "" sentry_capture: dsn: "" message: "" # No default (required) context: "" # No default (optional) extras: "" # No default (optional) tags: {} # No default (optional) environment: "" release: "" level: INFO transport_mode: async flush_timeout: 5s sampling_rate: 1 ``` ## [](#fields)Fields ### [](#context)`context` A mapping that must evaluate to an object-of-objects or `deleted()`. If this mapping produces a value, then it is set on a sentry event as additional context. **Type**: `string` ```yaml # Examples: context: 'root = {"order": {"product_id": "P93174", "quantity": 5}}' # --- context: root = deleted() ``` ### [](#dsn)`dsn` The DSN address to send sentry events to. If left empty, then SENTRY\_DSN is used. **Type**: `string` **Default**: `""` ### [](#environment)`environment` The environment to be sent with events. If left empty, then SENTRY\_ENVIRONMENT is used. **Type**: `string` **Default**: `""` ### [](#extras)`extras` A mapping that must evaluate to an object. If this mapping produces a value, then it is attached to the sentry event as a context named `extras`. (Prior to v4.x this populated the event’s deprecated Additional Data section, which the upstream sentry-go SDK removed.) Requires version 4.55.0 or later. **Type**: `string` ```yaml # Examples: extras: root.foo = "bar" # --- extras: root = this.without("password") ``` ### [](#flush_timeout)`flush_timeout` The duration to wait when closing the processor to flush any remaining enqueued events. **Type**: `string` **Default**: `5s` ### [](#level)`level` Sets the level on sentry events similar to logging levels. **Type**: `string` **Default**: `INFO` **Options**: `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL` ### [](#message)`message` A message to set on the sentry event This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: message: webhook event received # --- message: "failed to find product in database: ${! error() }" ``` ### [](#release)`release` The version of the code deployed to an environment. If left empty, then the Sentry client will attempt to detect the release from the environment. **Type**: `string` **Default**: `""` ### [](#sampling_rate)`sampling_rate` The rate at which events are sent to the server. A value of 0 disables capturing sentry events entirely. A value of 1 results in sending all events to Sentry. Any value in between results sending some percentage of events. **Type**: `float` **Default**: `1` ### [](#tags)`tags` Sets key/value string tags on an event. Unlike context, these are indexed and searchable on Sentry but have length limitations. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` ### [](#transport_mode)`transport_mode` Determines how events are sent. A sync transport will block when sending each event until a response is received from the Sentry server. The recommended async transport will enqueue events in a buffer and send them in the background. **Type**: `string` **Default**: `async` **Options**: `async`, `sync` --- # Page 288: slack_thread **URL**: https://docs.redpanda.com/connect/components/processors/slack_thread.md --- # slack_thread --- title: slack_thread latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/slack_thread page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/slack_thread.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/slack_thread.adoc description: Reads a Slack thread using the Slack API method conversations.replies. page-git-created-date: "2025-05-02" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Reads a Slack thread using the Slack API method conversations.replies. Introduced in version 4.52.0. This processor calls the [`conversations.replies`](https://api.slack.com/methods/conversations.replies) Slack API method and replaces the message content with a JSON array of all messages in the thread. #### Common ```yml processor: label: "" slack_thread: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" # No default (required) ``` #### Advanced ```yml processor: label: "" slack_thread: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" # No default (required) ``` ## [](#fields)Fields ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the correct permissions to read messages from the Slack channel specified in `channel_id`. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the Slack channel from which to read threads. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#thread_ts)`thread_ts` The timestamp of the parent message of the thread you want to read. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 289: sleep **URL**: https://docs.redpanda.com/connect/components/processors/sleep.md --- # sleep --- title: sleep latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sleep page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sleep.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sleep.adoc description: Sleep for a period of time specified as a duration string for each message. This processor will interpolate functions within the duration field, you can find a list of functions here. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Sleep for a period of time specified as a duration string for each message. This processor will interpolate functions within the `duration` field, you can find a list of functions [here](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). #### Common ```yml processor: label: "" sleep: duration: "" # No default (required) ``` #### Advanced ```yml processor: label: "" sleep: duration: "" # No default (required) ``` ## [](#fields)Fields ### [](#duration)`duration` The duration of time to sleep for each execution. This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 290: split **URL**: https://docs.redpanda.com/connect/components/processors/split.md --- # split --- title: split latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/split page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/split.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/split.adoc description: Breaks message batches (synonymous with multiple part messages) into smaller batches. The size of the resulting batches are determined either by a discrete size or, if the field byte_size is non-zero, then by total size in bytes (which ever limit is reached first). page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Breaks message batches (synonymous with multiple part messages) into smaller batches. The size of the resulting batches are determined either by a discrete size or, if the field `byte_size` is non-zero, then by total size in bytes (which ever limit is reached first). This processor is for breaking batches down into smaller ones. In order to break a single message out into multiple messages use the [`unarchive` processor](https://docs.redpanda.com/connect/components/processors/unarchive/). If there is a remainder of messages after splitting a batch the remainder is also sent as a single batch. For example, if your target size was 10, and the processor received a batch of 95 message parts, the result would be 9 batches of 10 messages followed by a batch of 5 messages. #### Common ```yml processor: label: "" split: size: 1 byte_size: 0 ``` #### Advanced ```yml processor: label: "" split: size: 1 byte_size: 0 ``` ## [](#fields)Fields ### [](#byte_size)`byte_size` An optional target of total message bytes. **Type**: `int` **Default**: `0` ### [](#size)`size` The target number of messages. **Type**: `int` **Default**: `1` --- # Page 291: sql_insert **URL**: https://docs.redpanda.com/connect/components/processors/sql_insert.md --- # sql_insert --- title: sql_insert latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sql_insert page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql_insert.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql_insert.adoc description: Inserts rows into an SQL database for each message, and leaves the message unchanged. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Inserts rows into an SQL database for each message, and leaves the message unchanged. Introduced in version 3.59.0. If the query fails to execute, the message remains unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). #### Common ```yml processor: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) ``` #### Advanced ```yml processor: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) prefix: "" # No default (optional) suffix: "" # No default (optional) options: [] # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) ``` If the insert fails to execute then the message will still remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml pipeline: processors: - sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ id, name, topic ] args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ## [](#dynamic-sql-operations)Dynamic SQL operations The `table` and `columns` fields are static strings that do not support Bloblang interpolation. For dynamic table names, dynamic column lists, DELETE operations, or any other SQL that `sql_insert` cannot express, use the [`sql_raw` processor](https://docs.redpanda.com/connect/components/processors/sql_raw/) instead. To use Bloblang interpolation inside ``sql_raw’s `query`` field, you must enable `unsafe_dynamic_query: true`. > ⚠️ **CAUTION** > > Interpolating unsanitized values into a query can introduce SQL injection risks. Always validate or sanitize the interpolated value beforehand. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of columns specified. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#columns)`columns[]` A list of columns to insert. **Type**: `array` ```yaml # Examples: columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#options)`options[]` A list of keyword options to add before the INTO clause of the query. Requires version 4.40.0 or later. **Type**: `array` ```yaml # Examples: options: - DELAYED - IGNORE ``` ### [](#prefix)`prefix` An optional prefix to prepend to the insert query (before INSERT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the insert query. **Type**: `string` ```yaml # Examples: suffix: ON CONFLICT (name) DO NOTHING ``` ### [](#table)`table` The table to insert to. **Type**: `string` ```yaml # Examples: table: foo ``` --- # Page 292: sql_raw **URL**: https://docs.redpanda.com/connect/components/processors/sql_raw.md --- # sql_raw --- title: sql_raw latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sql_raw page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql_raw.adoc description: Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. Introduced in version 3.65.0. If the query fails to execute, the message remains unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). #### Common ```yml processor: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) args_mapping: "" # No default (optional) exec_only: false # No default (optional) queries: [] # No default (optional) ``` #### Advanced ```yml processor: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) unsafe_dynamic_query: false args_mapping: "" # No default (optional) exec_only: false # No default (optional) queries: [] # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). For some scenarios where you might use this processor, see [Examples](#examples). ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#exec_only)`exec_only` Whether to discard the `query` result. Set to `true` to leave the message contents unchanged, which is useful when you are executing inserts, updates, and so on. By default, this is `false` for the last query and `true` for all previous queries, so only the last query’s result replaces the message contents. If you set this to `false` for a query other than the last, its result replaces the message contents and the `args_mapping` of each subsequent query runs against that result. **Type**: `bool` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#queries)`queries[]` A list of database statements to run in addition to your main `query`. If you specify multiple statements, they are executed within a single transaction. The output of the processor is always the result of the last query that runs, unless `exec_only` is used. Requires version 4.45.0 or later. **Type**: `array` ### [](#queries-args_mapping)`queries[].args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#queries-exec_only)`queries[].exec_only` Whether to discard the `query` result. Set to `true` to leave the message contents unchanged, which is useful when you are executing inserts, updates, and so on. By default, this is `false` for the last query and `true` for all previous queries, so only the last query’s result replaces the message contents. If you set this to `false` for a query other than the last, its result replaces the message contents and the `args_mapping` of each subsequent query runs against that result. **Type**: `bool` ### [](#queries-query)`queries[].query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); # --- query: SELECT * FROM footable WHERE user_id = $1; ``` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) in the query. Make sure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) The following example inserts rows into the table footable with the columns foo, bar and baz populated with values extracted from messages. ```yaml pipeline: processors: - sql_raw: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb query: "INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?);" args_mapping: '[ document.foo, document.bar, meta("kafka_topic") ]' exec_only: true ``` ### [](#table-query-postgresql)Table Query (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message field `user.id`. A [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`. ```yaml pipeline: processors: - branch: processors: - sql_raw: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable query: "SELECT * FROM footable WHERE user_id = $1;" args_mapping: '[ this.user.id ]' result_map: 'root.foo_rows = this' ``` ### [](#dynamically-creating-tables-postgresql)Dynamically Creating Tables (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message field `user.id`. A [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`. ```yaml pipeline: processors: - mapping: | root = this # Prevent SQL injection when using unsafe_dynamic_query meta table_name = "\"" + metadata("table_name").replace_all("\"", "\"\"") + "\"" - sql_raw: driver: postgres dsn: postgres://localhost/postgres unsafe_dynamic_query: true queries: - query: | CREATE TABLE IF NOT EXISTS ${!metadata("table_name")} (id varchar primary key, document jsonb); - query: | INSERT INTO ${!metadata("table_name")} (id, document) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET document = EXCLUDED.document; args_mapping: | root = [ this.id, this.document.string() ] ``` --- # Page 293: sql_select **URL**: https://docs.redpanda.com/connect/components/processors/sql_select.md --- # sql_select --- title: sql_select latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sql_select page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql_select.adoc description: Runs an SQL select query against a database and returns the result as an array of objects, one for each row returned, containing a key for each column queried and its value. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Runs an SQL select query against a database and returns the result as an array of objects, one for each row returned, containing a key for each column queried and its value. Introduced in version 3.59.0. If the query fails to execute, the message remains unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). #### Common ```yml processor: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processor: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: 0 # No default (optional) ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#examples)Examples ### [](#table-query-postgresql)Table Query (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message `user.id`. A [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`: ```yaml pipeline: processors: - branch: processors: - sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] where: user_id = ? args_mapping: '[ this.user.id ]' result_map: 'root.foo_rows = this' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that evaluates to an array of values, one for each placeholder in the `where` field, in order. The values are passed to the driver as query arguments. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ```yaml # Examples: columns: - "*" # --- columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following table lists the supported drivers and their DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | | hana | hdb://[user[:password]@]host[:port][?param1=value1&…​] | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake/v2#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: | CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#prefix)`prefix` Optional SQL text to add before the `SELECT` keyword, such as a `WITH` clause. **Type**: `string` ### [](#suffix)`suffix` Optional SQL text to append after the generated `SELECT` statement, such as an `ORDER BY`, `LIMIT`, or locking clause. **Type**: `string` ### [](#table)`table` The table to query. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Always use question marks for placeholders. They are converted to dollar syntax (`$1`) when the `postgres`, `pgx`, or `clickhouse` driver is used, and to colon syntax (`:1`) when the `oracle` or `gocosmos` driver is used. **Type**: `string` ```yaml # Examples: where: meow = ? and woof = ? # --- where: user_id = ? ``` --- # Page 294: sql **URL**: https://docs.redpanda.com/connect/components/processors/sql.md --- # sql --- title: sql latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sql page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql.adoc description: Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **WARNING: Deprecated** > > Deprecated > > This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. Introduced in version 3.65.0. #### Common ```yml processor: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) result_codec: none ``` #### Advanced ```yml processor: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) unsafe_dynamic_query: false args_mapping: "" # No default (optional) result_codec: none ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#alternatives)Alternatives For basic inserts or select queries use either the [`sql_insert`](https://docs.redpanda.com/connect/components/processors/sql_insert/) or the [`sql_select`](https://docs.redpanda.com/connect/components/processors/sql_select/) processor. For more complex queries use the [`sql_raw`](https://docs.redpanda.com/connect/components/processors/sql_raw/) processor. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the `query` field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#data_source_name)`data_source_name` Data source name. **Type**: `string` ### [](#driver)`driver` The database driver to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks`, `hana` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). The following table shows the placeholder style for each driver: | Driver | Placeholder style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mssql | Question mark (?) | | mysql | Question mark (?) | | oracle | Colon (:) | | pgx | Dollar sign ($) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` ### [](#result_codec)`result_codec` Result codec. **Type**: `string` **Default**: `none` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries) in the query. Make sure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` --- # Page 295: string_split **URL**: https://docs.redpanda.com/connect/components/processors/string_split.md --- # string_split --- title: string_split latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/string_split page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/string_split.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/string_split.adoc description: Splits a string by a delimiter into an array. Generally, using bloblang's split method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. page-git-created-date: "2026-04-03" page-git-modified-date: "2026-10-10" --- Splits a string by a delimiter into an array. Generally, using bloblang’s `split` method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. Introduced in version 4.86.0. #### Common ```yml processor: label: "" string_split: delimiter: |+ empty_as_null: false ``` #### Advanced ```yml processor: label: "" string_split: delimiter: |+ emit_bytes: false empty_as_null: false ``` ## [](#fields)Fields ### [](#delimiter)`delimiter` The delimiter to split the string by. **Type**: `string` **Default**: `"\n"` ### [](#emit_bytes)`emit_bytes` When true, the output will be bloblang bytes instead of strings. **Type**: `bool` **Default**: `false` ### [](#empty_as_null)`empty_as_null` When true, empty strings resulting from the split are converted to null. **Type**: `bool` **Default**: `false` --- # Page 296: subprocess **URL**: https://docs.redpanda.com/connect/components/processors/subprocess.md --- # subprocess --- title: subprocess latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/subprocess page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/subprocess.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/subprocess.adoc description: Executes a command as a subprocess and, for each message, will pipe its contents to the stdin stream of the process followed by a newline. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a command as a subprocess and, for each message, will pipe its contents to the stdin stream of the process followed by a newline. > 📝 **NOTE** > > This processor keeps the subprocess alive and requires very specific behavior from the command executed. If you wish to simply execute a command for each message take a look at the [`command` processor](https://docs.redpanda.com/connect/components/processors/command/) instead. The subprocess must then either return a line over stdout or stderr. If a response is returned over stdout then its contents will replace the message. If a response is instead returned from stderr it will be logged and the message will continue unchanged and will be [marked as failed](https://docs.redpanda.com/connect/configuration/error_handling/). Rather than separating data by a newline it’s possible to specify alternative [`codec_send`](#codec_send) and [`codec_recv`](#codec_recv) values, which allow binary messages to be encoded for logical separation. The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory. The field `max_buffer` defines the maximum response size able to be read from the subprocess. This value should be set significantly above the real expected maximum response size. ## [](#subprocess-requirements)Subprocess requirements It is required that subprocesses flush their stdout and stderr pipes for each line. Redpanda Connect will attempt to keep the process alive for as long as the pipeline is running. If the process exits early it will be restarted. ## [](#messages-containing-line-breaks)Messages containing line breaks If a message contains line breaks each line of the message is piped to the subprocess and flushed, and a response is expected from the subprocess before another line is fed in. ### Common ```yml processor: label: "" subprocess: name: "" # No default (required) args: [] ``` ### Advanced ```yml processor: label: "" subprocess: name: "" # No default (required) args: [] max_buffer: 65536 codec_send: lines codec_recv: lines ``` ## [](#fields)Fields ### [](#args)`args[]` A list of arguments to provide the command. **Type**: `array` **Default**: `[]` ### [](#codec_recv)`codec_recv` Determines how messages read from the subprocess are decoded, which allows them to be logically separated. Requires version 3.37.0 or later. **Type**: `string` **Default**: `lines` **Options**: `lines`, `length_prefixed_uint32_be`, `netstring` ### [](#codec_send)`codec_send` Determines how messages written to the subprocess are encoded, which allows them to be logically separated. Requires version 3.37.0 or later. **Type**: `string` **Default**: `lines` **Options**: `lines`, `length_prefixed_uint32_be`, `netstring` ### [](#max_buffer)`max_buffer` The maximum expected response size. **Type**: `int` **Default**: `65536` ### [](#name)`name` The command to execute as a subprocess. **Type**: `string` ```yaml # Examples: name: cat # --- name: sed # --- name: awk ``` --- # Page 297: switch **URL**: https://docs.redpanda.com/connect/components/processors/switch.md --- # switch --- title: switch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/switch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/switch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/switch.adoc description: Conditionally processes messages based on their contents. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Conditionally processes messages based on their contents. For each switch case a [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) is checked and, if the result is true (or the check is empty) the child processors are executed on the message. ### Common ```yml processor: label: "" switch: - check: "" processors: [] ``` ### Advanced ```yml processor: label: "" switch: - check: "" processors: [] fallthrough: false continue: false ``` ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should have the processors of this case executed on it. If left empty the case always passes. If the check mapping throws an error the message will be flagged [as having failed](https://docs.redpanda.com/connect/configuration/error_handling/) and will not be tested against any other cases. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") ``` ### [](#continue)`continue` Indicates whether, if this case passes for a message, the next case should also be tested. Unlike `fallthrough`, which skips the next case’s check, `continue` will evaluate the next case’s condition before executing. **Type**: `bool` **Default**: `false` ### [](#fallthrough)`fallthrough` Indicates whether, if this case passes for a message, the next case should also be executed without checking its condition. **Type**: `bool` **Default**: `false` ### [](#processors)`processors[]` A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to execute on a message. **Type**: `array` **Default**: `[]` ## [](#examples)Examples ### [](#ignore-george)Ignore George We have a system where we’re counting a metric for all messages that pass through our system. However, occasionally we get messages from George that we don’t care about. For George’s messages we want to instead emit a metric that gauges how angry he is about being ignored and then we drop it. ```yaml pipeline: processors: - switch: - check: this.user.name.first != "George" processors: - metric: type: counter name: MessagesWeCareAbout - processors: - metric: type: gauge name: GeorgesAnger value: ${! json("user.anger") } - mapping: root = deleted() ``` ## [](#batching)Batching When a switch processor executes on a [batch of messages](https://docs.redpanda.com/connect/configuration/batching/) they are checked individually and can be matched independently against cases. During processing the messages matched against a case are processed as a batch, although the ordering of messages during case processing cannot be guaranteed to match the order as received. At the end of switch processing the resulting batch will follow the same ordering as the batch was received. If any child processors have split or otherwise grouped messages this grouping will be lost as the result of a switch is always a single batch. In order to perform conditional grouping and/or splitting use the [`group_by` processor](https://docs.redpanda.com/connect/components/processors/group_by/). --- # Page 298: sync_response **URL**: https://docs.redpanda.com/connect/components/processors/sync_response.md --- # sync_response --- title: sync_response latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/sync_response page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sync_response.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sync_response.adoc description: Adds the payload in its current state as a synchronous response to the input source, where it is dealt with according to that specific input type. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Adds the payload in its current state as a synchronous response to the input source, where it is dealt with according to that specific input type. ## Common ```yml processor: label: "" sync_response: {} ``` ## Advanced ```yml processor: label: "" sync_response: {} ``` For most inputs this mechanism is ignored entirely, in which case the sync response is dropped without penalty. It is therefore safe to use this processor even when combining input types that might not have support for sync responses. An example of an input able to utilize this is the `http_server`. For more information please read [synchronous responses](https://docs.redpanda.com/connect/guides/sync_responses/). --- # Page 299: text_chunker **URL**: https://docs.redpanda.com/connect/components/processors/text_chunker.md --- # text_chunker --- title: text_chunker latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/text_chunker page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/text_chunker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/text_chunker.adoc description: A processor that allows chunking and splitting text based on some strategy. Usually used for creating vector embeddings of large documents. page-git-created-date: "2025-05-02" page-git-modified-date: "2026-10-10" --- Breaks down text-based message content into manageable chunks using a configurable strategy. This processor is ideal for creating vector embeddings of large text documents. Introduced in version 4.51.0. #### Common ```yml processor: label: "" text_chunker: strategy: "" # No default (required) chunk_size: 512 chunk_overlap: 100 separators: - "\n\n" - "\n" - " " - "" length_measure: runes include_code_blocks: false keep_reference_links: false ``` #### Advanced ```yml processor: label: "" text_chunker: strategy: "" # No default (required) chunk_size: 512 chunk_overlap: 100 separators: - "\n\n" - "\n" - " " - "" length_measure: runes token_encoding: "" # No default (optional) allowed_special: [] disallowed_special: - "all" include_code_blocks: false keep_reference_links: false ``` ## [](#fields)Fields ### [](#allowed_special)`allowed_special[]` A list of special tokens to include in the output from this processor. **Type**: `array` **Default**: `[]` ### [](#chunk_overlap)`chunk_overlap` The number of characters duplicated in adjacent chunks of text. **Type**: `int` **Default**: `100` ### [](#chunk_size)`chunk_size` The maximum size of each chunk, using the selected `length_measure`. **Type**: `int` **Default**: `512` ### [](#disallowed_special)`disallowed_special[]` A list of special tokens to exclude from the output of this processor. **Type**: `array` **Default**: ```yaml - "all" ``` ### [](#include_code_blocks)`include_code_blocks` When set to `true`, this processor includes code blocks in the output. **Type**: `bool` **Default**: `false` ### [](#keep_reference_links)`keep_reference_links` When set to `true`, this processor includes reference links in the output. **Type**: `bool` **Default**: `false` ### [](#length_measure)`length_measure` Choose a method to measure the length of a string. **Type**: `string` **Default**: `runes` | Option | Summary | | --- | --- | | graphemes | Use unicode graphemes to determine the length of a string. | | runes | Use the number of codepoints to determine the length of a string. | | token | Use the number of tokens (using the token_encoding tokenizer) to determine the length of a string. | | utf8 | Determine the length of text using the number of utf8 bytes. | ### [](#separators)`separators[]` A list of strings to use as separators between chunks when the `strategy` field is set to `recursive_character`. By default, the following separators are tried in turn: double newlines (`\n\n`), single newlines (`\n`), spaces (`" “), and the empty string (”"`). **Type**: `array` **Default**: ```yaml - "\n\n" - "\n" - " " - "" ``` ### [](#strategy)`strategy` Choose a strategy for breaking content down into chunks. **Type**: `string` | Option | Summary | | --- | --- | | markdown | Split text by markdown headers. | | recursive_character | Split text recursively by characters (defined in separators). | | token | Split text by tokens. | ### [](#token_encoding)`token_encoding` The type of encoding to use for tokenization. **Type**: `string` ```yaml # Examples: token_encoding: cl100k_base # --- token_encoding: r50k_base ``` --- # Page 300: try_catch **URL**: https://docs.redpanda.com/connect/components/processors/try_catch.md --- # try_catch --- title: try_catch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/try_catch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/try_catch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/try_catch.adoc description: Executes a list of child processors on each message and, if any of them fail, executes a separate list of catch processors to recover from or react to the error. page-git-created-date: "2026-07-07" page-git-modified-date: "2026-10-10" --- Executes a list of child `processors` on each message and, if any of them fail, executes a separate list of `catch` processors to recover from or react to the error. This processor combines the behavior of the [`try`](https://docs.redpanda.com/connect/components/processors/try/) and [`catch`](https://docs.redpanda.com/connect/components/processors/catch/) processors into a single block with an explicit recovery path. Because it contains both the fallible step and its recovery within a single processor, it is the recommended way to handle expected errors when strict error handling (`error_handling.strict`) is enabled. Each message of a batch is processed individually. The `processors` field is executed with "try" semantics: as soon as a processor fails for a given message the remaining `processors` are skipped for that message. Any message that failed is then routed to the `catch` processors. Before they run, the failure is moved off the message: it is stored as a structured object in a metadata field (see `error_metadata`, `error` by default) and the message’s failure flag is **cleared**. The error is therefore available to recovery logic as an ordinary variable rather than as a message property. The object contains: - `what`: the error message. - `name`: the name of the component that failed (when known). - `label`: the label of the component that failed (when set). - `path`: the dot-path of the component that failed (when known). So a recovery mapping reads the failure with, for example, `@error.what` (equivalent to `meta("error").what`). Because the flag is cleared, the `catch` processors run under the normal error semantics, including strict, so a _new_ failure raised while recovering is treated as a fresh error and is not silently tolerated. > 📝 **NOTE** > > Because the failure flag is cleared before the `catch` processors run, the [`error`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error) and `error_source_*` functions do not report the original failure within the `catch` block; use the metadata object instead. An empty or omitted `catch` simply records the error in metadata and clears the flag (the failure is swallowed). ```yaml pipeline: processors: - try_catch: processors: - resource: foo - resource: bar catch: - mutation: 'root = "failed to process: " + @error.what' ``` In the example above, if either `foo` or `bar` fails for a message then the `mutation` is applied to that message, replacing its contents with a description of the error (read from the metadata object), and the message continues downstream without a failure flag. More information about error handling can be found in [Error Handling](https://docs.redpanda.com/connect/configuration/error_handling/). #### Common ```yml processor: label: "" try_catch: processors: [] catch: [] error_metadata: error ``` #### Advanced ```yml processor: label: "" try_catch: processors: [] catch: [] error_metadata: error ``` ## [](#fields)Fields ### [](#catch)`catch[]` A list of processors to execute on each message that failed one of the `processors` above. The message is no longer flagged as failed when these run; the error is available as an object in the metadata field named by `error_metadata` (for example, `@error.what`). When omitted or empty the error is recorded in metadata and the flag is cleared (the failure is swallowed). **Type**: `array` **Default**: `[]` ### [](#error_metadata)`error_metadata` The metadata key under which the caught error is stored, as an object with a `what` field (the error message) plus `name`, `label` and `path` fields describing the component that failed, before the `catch` processors are executed. **Type**: `string` **Default**: `error` ### [](#processors)`processors[]` A list of processors to execute on each message. If a processor fails for a given message the remaining processors in this list are skipped for that message, and the message is routed to the `catch` processors. **Type**: `array` **Default**: `[]` --- # Page 301: try **URL**: https://docs.redpanda.com/connect/components/processors/try.md --- # try --- title: try latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/try page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/try.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/try.adoc description: Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared). page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared). ### Common ```yml processor: label: "" try: [] ``` ### Advanced ```yml processor: label: "" try: [] ``` This processor behaves similarly to the [`for_each`](https://docs.redpanda.com/connect/components/processors/for_each/) processor, where a list of child processors are applied to individual messages of a batch. However, if a message has failed any prior processor (before or during the try block) then that message will skip all following processors. For example, with the following config: ```yaml pipeline: processors: - resource: foo - try: - resource: bar - resource: baz - resource: buz ``` If the processor `bar` fails for a particular message, that message will skip the processors `baz` and `buz`. Similarly, if `bar` succeeds but `baz` does not then `buz` will be skipped. If the processor `foo` fails for a message then none of `bar`, `baz` or `buz` are executed on that message. This processor is useful for when child processors depend on the successful output of previous processors. This processor can be followed with a [catch](https://docs.redpanda.com/connect/components/processors/catch/) processor for defining child processors to be applied only to failed messages. More information about error handing can be found in [Error Handling](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#nest-within-a-catch-block)Nest within a catch block In some cases it might be useful to nest a try block within a catch block, since the [`catch` processor](https://docs.redpanda.com/connect/components/processors/catch/) only clears errors _after_ executing its child processors this means a nested try processor will not execute unless the errors are explicitly cleared beforehand. This can be done by inserting an empty catch block before the try block like as follows: ```yaml pipeline: processors: - resource: foo - catch: - log: level: ERROR message: "Foo failed due to: ${! error() }" - catch: [] # Clear prior error - try: - resource: bar - resource: baz ``` --- # Page 302: unarchive **URL**: https://docs.redpanda.com/connect/components/processors/unarchive.md --- # unarchive --- title: unarchive latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/unarchive page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/unarchive.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/unarchive.adoc description: Unarchives messages according to the selected archive format into multiple messages within a batch. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Unarchives messages according to the selected archive format into multiple messages within a [batch](https://docs.redpanda.com/connect/configuration/batching/). When a message is unarchived the new messages replace the original message in the batch. Messages that are selected but fail to unarchive (invalid format) will remain unchanged in the message batch but will be flagged as having failed, allowing you to [error handle them](https://docs.redpanda.com/connect/configuration/error_handling/). Unarchived messages are kept in the same batch. To process each unarchived message individually, follow this processor with a [`split` processor](https://docs.redpanda.com/connect/components/processors/split/). ## [](#metadata)Metadata The metadata found on the messages handled by this processor will be copied into the resulting messages. For the unarchive formats that contain file information (tar, zip), a metadata field is also added to each message called `archive_filename` with the extracted filename. ### Common ```yml processor: label: "" unarchive: format: "" # No default (required) ``` ### Advanced ```yml processor: label: "" unarchive: format: "" # No default (required) ``` ## [](#fields)Fields ### [](#format)`format` The unarchiving format to apply. **Type**: `string` | Option | Summary | | --- | --- | | binary | Extract messages from a binary blob format. | | csv | Attempt to parse the message as a csv file (header required) and for each row in the file expands its contents into a json object in a new message. | | csv:x | Attempt to parse the message as a csv file (header required) and for each row in the file expands its contents into a json object in a new message using a custom delimiter. The custom delimiter must be a single character. For example, the format "csv:\t" would consume a tab delimited file. | | json_array | Attempt to parse a message as a JSON array, and extract each element into its own message. | | json_documents | Attempt to parse a message as a stream of concatenated JSON documents. Each parsed document is expanded into a new message. | | json_map | Attempt to parse the message as a JSON map and for each element of the map expands its contents into a new message. A metadata field is added to each message called archive_key with the relevant key from the top-level map. | | lines | Extract the lines of a message each into their own message. | | tar | Extract messages from a unix standard tape archive. | | zip | Extract messages from a zip file. | --- # Page 303: wasm **URL**: https://docs.redpanda.com/connect/components/processors/wasm.md --- # wasm --- title: wasm latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/wasm page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/wasm.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/wasm.adoc description: Executes a function exported by a WASM module for each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a function exported by a WASM module for each message. Introduced in version 4.11.0. This processor uses [Wazero](https://github.com/tetratelabs/wazero) to execute a WASM module (with support for WASI), calling a specific function for each message being processed. From within the WASM module it is possible to query and mutate the message being processed via a suite of functions exported to the module. This ecosystem is delicate as WASM doesn’t have a single clearly defined way to pass strings back and forth between the host and the module. In order to remedy this we’re gradually working on introducing libraries and examples for multiple languages which can be found in [the codebase](https://github.com/redpanda-data/benthos/tree/main/public/wasm/README.md). These examples, as well as the processor itself, is a work in progress. ## [](#parallelism)Parallelism It’s not currently possible to execute a single WASM runtime across parallel threads with this processor. Therefore, in order to support parallel processing this processor implements pooling of module runtimes. Ideally your WASM module shouldn’t depend on any global state, but if it does then you need to ensure the processor [is only run on a single thread](https://docs.redpanda.com/connect/configuration/processing_pipelines/). ### Common ```yml processor: label: "" wasm: module_path: "" # No default (required) function: process ``` ### Advanced ```yml processor: label: "" wasm: module_path: "" # No default (required) function: process ``` ## [](#fields)Fields ### [](#function)`function` The name of the function exported by the target WASM module to run for each message. **Type**: `string` **Default**: `process` ### [](#module_path)`module_path` The path of the WASM module file to load, which must export the function named in `function`. The file is read once, when the processor is created. **Type**: `string` --- # Page 304: while **URL**: https://docs.redpanda.com/connect/components/processors/while.md --- # while --- title: while latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/while page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/while.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/while.adoc description: A processor that checks a Bloblang query against each batch of messages and executes child processors on them for as long as the query resolves to true. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A processor that checks a [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) against each batch of messages and executes child processors on them for as long as the query resolves to true. The field `at_least_once`, if true, ensures that the child processors are always executed at least one time (like a do .. while loop.) The field `max_loops`, if greater than zero, caps the number of loops for a message batch to this value. If following a loop execution the number of messages in a batch is reduced to zero the loop is exited regardless of the condition result. If following a loop execution there are more than 1 message batches the query is checked against the first batch only. The conditions of this processor are applied across entire message batches. You can find out more about batching [in this doc](https://docs.redpanda.com/connect/configuration/batching/). #### Common ```yml processor: label: "" while: at_least_once: false check: "" processors: [] # No default (required) ``` #### Advanced ```yml processor: label: "" while: at_least_once: false max_loops: 0 check: "" processors: [] # No default (required) ``` ## [](#fields)Fields ### [](#at_least_once)`at_least_once` Whether to always run the child processors at least one time. **Type**: `bool` **Default**: `false` ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that should return a boolean value indicating whether the while loop should execute again. **Type**: `string` **Default**: `""` ```yaml # Examples: check: errored() # --- check: this.urls.unprocessed.length() > 0 ``` ### [](#max_loops)`max_loops` An optional maximum number of loops to execute. Helps protect against accidentally creating infinite loops. **Type**: `int` **Default**: `0` ### [](#processors)`processors[]` A list of child processors to execute on each loop. **Type**: `array` --- # Page 305: workflow **URL**: https://docs.redpanda.com/connect/components/processors/workflow.md --- # workflow --- title: workflow latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/workflow page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/workflow.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/workflow.adoc description: Executes a topology of branch processors, performing them in parallel where possible. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Executes a topology of [`branch` processors](https://docs.redpanda.com/connect/components/processors/branch/), performing them in parallel where possible. ### Common ```yml processor: label: "" workflow: meta_path: meta.workflow order: [] branches: : request_map: "" processors: [] # No default (required) result_map: "" ``` ### Advanced ```yml processor: label: "" workflow: meta_path: meta.workflow order: [] branch_resources: [] branches: : request_map: "" processors: [] # No default (required) result_map: "" ``` ## [](#why-use-a-workflow)Why use a workflow ### [](#performance)Performance Most of the time the best way to compose processors is also the simplest, just configure them in series. This is because processors are often CPU bound, low-latency, and you can gain vertical scaling by increasing the number of processor pipeline threads, allowing Redpanda Connect to process [multiple messages in parallel](https://docs.redpanda.com/connect/configuration/processing_pipelines/). However, some processors, such as [`http`](https://docs.redpanda.com/connect/components/processors/http/), [`aws_lambda`](https://docs.redpanda.com/connect/components/processors/aws_lambda/) or [`cache`](https://docs.redpanda.com/connect/components/processors/cache/), interact with external services and therefore spend most of their time waiting for a response. These processors tend to be high-latency and low CPU activity, which causes messages to process slowly. When a processing pipeline contains multiple network processors that aren’t dependent on each other we can benefit from performing these processors in parallel for each individual message, reducing the overall message processing latency. ### [](#simplifying-processor-topology)Simplifying processor topology A workflow is often expressed as a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of processing stages, where each stage can result in N possible next stages, until finally the flow ends at an exit node. For example, if we had processing stages A, B, C and D, where stage A could result in either stage B or C being next, always followed by D, it might look something like this: ```text /--> B --\ A --| |--> D \--> C --/ ``` This flow would be easy to express in a standard Redpanda Connect config, we could simply use a [`switch` processor](https://docs.redpanda.com/connect/components/processors/switch/) to route to either B or C depending on a condition on the result of A. However, this method of flow control quickly becomes unfeasible as the DAG gets more complicated, imagine expressing this flow using switch processors: ```text /--> B -------------|--> D / / A --| /--> E --| \--> C --| \ \----------|--> F ``` And imagine doing so knowing that the diagram is subject to change over time. Yikes! Instead, with a workflow we can either trust it to automatically resolve the DAG or express it manually as simply as `order: [ [ A ], [ B, C ], [ E ], [ D, F ] ]`, and the conditional logic for determining if a stage is executed is defined as part of the branch itself. ## [](#examples)Examples ### [](#automatic-ordering)Automatic Ordering When the field `order` is omitted a best attempt is made to determine a dependency tree between branches based on their request and result mappings. In the following example the branches foo and bar will be executed first in parallel, and afterwards the branch baz will be executed. ```yaml pipeline: processors: - workflow: meta_path: meta.workflow branches: foo: request_map: 'root = ""' processors: - http: url: TODO result_map: 'root.foo = this' bar: request_map: 'root = this.body' processors: - aws_lambda: function: TODO result_map: 'root.bar = this' baz: request_map: | root.fooid = this.foo.id root.barstuff = this.bar.content processors: - cache: resource: TODO operator: set key: ${! json("fooid") } value: ${! json("barstuff") } ``` ### [](#conditional-branches)Conditional Branches Branches of a workflow are skipped when the `request_map` assigns `deleted()` to the root. In this example the branch A is executed when the document type is "foo", and branch B otherwise. Branch C is executed afterwards and is skipped unless either A or B successfully provided a result at `tmp.result`. ```yaml pipeline: processors: - workflow: branches: A: request_map: | root = if this.document.type != "foo" { deleted() } processors: - http: url: TODO result_map: 'root.tmp.result = this' B: request_map: | root = if this.document.type == "foo" { deleted() } processors: - aws_lambda: function: TODO result_map: 'root.tmp.result = this' C: request_map: | root = if this.tmp.result != null { deleted() } processors: - http: url: TODO_SOMEWHERE_ELSE result_map: 'root.tmp.result = this' ``` ### [](#resources)Resources The `order` field can be used in order to refer to [branch processor resources](#resources), this can sometimes make your pipeline configuration cleaner, as well as allowing you to reuse branch configurations in order places. It’s also possible to mix and match branches configured within the workflow and configured as resources. ```yaml pipeline: processors: - workflow: order: [ [ foo, bar ], [ baz ] ] branches: bar: request_map: 'root = this.body' processors: - aws_lambda: function: TODO result_map: 'root.bar = this' processor_resources: - label: foo branch: request_map: 'root = ""' processors: - http: url: TODO result_map: 'root.foo = this' - label: baz branch: request_map: | root.fooid = this.foo.id root.barstuff = this.bar.content processors: - cache: resource: TODO operator: set key: ${! json("fooid") } value: ${! json("barstuff") } ``` ## [](#fields)Fields ### [](#branch_resources)`branch_resources[]` An optional list of [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) names that are configured as [Resources](#resources). These resources will be included in the workflow with any branches configured inline within the [`branches`](#branches) field. The order and parallelism in which branches are executed is automatically resolved based on the mappings of each branch. When using resources with an explicit order it is not necessary to list resources in this field. Requires version 3.38.0 or later. **Type**: `array` **Default**: `[]` ### [](#branches)`branches` An object of named [`branch` processors](https://docs.redpanda.com/connect/components/processors/branch/) that make up the workflow. The order and parallelism in which branches are executed can either be made explicit with the field `order`, or if omitted an attempt is made to automatically resolve an ordering based on the mappings of each branch. **Type**: `object` **Default**: `{}` ### [](#branches-processors)`branches.processors[]` A list of processors to apply to mapped requests. When processing message batches the resulting batch must match the size and ordering of the input batch, therefore filtering, grouping should not be performed within these processors. **Type**: `array` ### [](#branches-request_map)`branches.request_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that describes how to create a request payload suitable for the child processors of this branch. If left empty then the branch will begin with an exact copy of the origin message (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: request_map: |- root = { "id": this.doc.id, "content": this.doc.body.text } # --- request_map: |- root = if this.type == "foo" { this.foo.request } else { deleted() } ``` ### [](#branches-result_map)`branches.result_map` A [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that describes how the resulting messages from branched processing should be mapped back into the original payload. If left empty the origin message will remain unchanged (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: result_map: |- meta foo_code = metadata("code") root.foo_result = this # --- result_map: |- meta = metadata() root.bar.body = this.body root.bar.id = this.user.id # --- result_map: root.raw_result = content().string() # --- result_map: |- root.enrichments.foo = if metadata("request_failed") != null { throw(metadata("request_failed")) } else { this } # --- result_map: |- # Retain only the updated metadata fields which were present in the origin message meta = metadata().filter(v -> @.get(v.key) != null) ``` ### [](#meta_path)`meta_path` A [dot path](https://docs.redpanda.com/connect/configuration/field_paths/) indicating where to store and reference [structured metadata](#structured-metadata) about the workflow execution. **Type**: `string` **Default**: `meta.workflow` ### [](#order)`order[][]` An explicit declaration of branch ordered tiers, which describes the order in which parallel tiers of branches should be executed. Branches should be identified by the name as they are configured in the field `branches`. It’s also possible to specify branch processors configured [as a resource](#resources). **Type**: `array>` **Default**: `[]` ```yaml # Examples: order: - - foo - bar - - baz # --- order: - - foo - - bar - - baz ``` ## [](#structured-metadata)Structured metadata When the field `meta_path` is non-empty the workflow processor creates an object describing which workflows were successful, skipped or failed for each message and stores the object within the message at the end. The object is of the following form: ```json { "succeeded": [ "foo" ], "skipped": [ "bar" ], "failed": { "baz": "the error message from the branch" } } ``` If a message already has a meta object at the given path when it is processed then the object is used in order to determine which branches have already been performed on the message (or skipped) and can therefore be skipped on this run. This is a useful pattern when replaying messages that have failed some branches previously. For example, given the above example object the branches foo and bar would automatically be skipped, and baz would be reattempted. The previous meta object will also be preserved in the field `.previous` when the new meta object is written, preserving a full record of all workflow executions. If a field `.apply` exists in the meta object for a message and is an array then it will be used as an explicit list of stages to apply, all other stages will be skipped. ## [](#resources-2)Resources It’s common to configure processors (and other components) [as resources](https://docs.redpanda.com/connect/configuration/resources/) in order to keep the pipeline configuration cleaner. With the workflow processor you can include branch processors configured as resources within your workflow either by specifying them by name in the field `order`, if Redpanda Connect doesn’t find a branch within the workflow configuration of that name it’ll refer to the resources. Alternatively, if you do not wish to have an explicit ordering, you can add resource names to the field `branch_resources` and they will be included in the workflow with automatic DAG resolution along with any branches configured in the `branches` field. ### [](#resource-error-conditions)Resource error conditions There are two error conditions that could potentially occur when resources included in your workflow are mutated, and if you are planning to mutate resources in your workflow it is important that you understand them. The first error case is that a resource in the workflow is removed and not replaced, when this happens the workflow will still be executed but the individual branch will fail. This should only happen if you explicitly delete a branch resource, as any mutation operation will create the new resource before removing the old one. The second error case is when automatic DAG resolution is being used and a resource in the workflow is changed in a way that breaks the DAG (circular dependencies, etc). When this happens it is impossible to execute the workflow and therefore the processor will fail, which is possible to capture and handle using [standard error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). ## [](#error-handling)Error handling The recommended approach to handle failures within a workflow is to query against the [structured metadata](#structured-metadata) it provides, as it provides granular information about exactly which branches failed and which ones succeeded and therefore aren’t necessary to perform again. For example, if our meta object is stored at the path `meta.workflow` and we wanted to check whether a message has failed for any branch we can do that using a [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) like `this.meta.workflow.failed.length() | 0 > 0`, or to check whether a specific branch failed we can use `this.exists("meta.workflow.failed.foo")`. However, if structured metadata is disabled by setting the field `meta_path` to empty then the workflow processor instead adds a general error flag to messages when any executed branch fails. In this case it’s possible to handle failures using [standard error handling patterns](https://docs.redpanda.com/connect/configuration/error_handling/). --- # Page 306: xml **URL**: https://docs.redpanda.com/connect/components/processors/xml.md --- # xml --- title: xml latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processors/xml page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/xml.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/xml.adoc description: Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value. #### Common ```yml processor: label: "" xml: operator: "" cast: false ``` #### Advanced ```yml processor: label: "" xml: operator: "" cast: false ``` ## [](#operators)Operators ### [](#to_json)`to_json` Converts an XML document into a JSON structure, where elements appear as keys of an object according to the following rules: - If an element contains attributes they are parsed by prefixing a hyphen, `-`, to the attribute label. - If the element is a simple element and has attributes, the element value is given the key `#text`. - XML comments, directives, and process instructions are ignored. - When elements are repeated the resulting JSON value is an array. - XML namespaces are stripped from element and attribute names, and namespace declarations (`xmlns`) are omitted. For example, given the following XML: ```xml This is a title This is a description foo1 foo2 foo3 ``` The resulting JSON structure would look like this: ```json { "root":{ "title":"This is a title", "description":{ "#text":"This is a description", "-tone":"boring" }, "elements":[ {"#text":"foo1","-id":"1"}, {"#text":"foo2","-id":"2"}, "foo3" ] } } ``` With cast set to true, the resulting JSON structure would look like this: ```json { "root":{ "title":"This is a title", "description":{ "#text":"This is a description", "-tone":"boring" }, "elements":[ {"#text":"foo1","-id":1}, {"#text":"foo2","-id":2}, "foo3" ] } } ``` ## [](#fields)Fields ### [](#cast)`cast` Whether to try to cast values that are numbers and booleans to the right type. Default: all values are strings. **Type**: `bool` **Default**: `false` ### [](#operator)`operator` An XML [operation](#operators) to apply to messages. **Type**: `string` **Default**: `""` **Options**: `to_json` --- # Page 307: Rate Limits **URL**: https://docs.redpanda.com/connect/components/rate_limits/about.md --- # Rate Limits --- title: Rate Limits latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: rate_limits/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rate_limits/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/rate_limits/about.adoc description: Learn how rate limits work in Redpanda Connect. A rate limit is a named resource that limits use of a shared resource across parallel components. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- A rate limit is a strategy for limiting the usage of a shared resource across parallel components in a Redpanda Connect instance, or potentially across multiple instances. They are configured as a resource: ```yaml rate_limit_resources: - label: foobar local: count: 500 interval: 1s ``` And most components that hit external services have a field `rate_limit` for specifying a rate limit resource to use, identified by the `label` field. For example, if we wanted to use our `foobar` rate limit with an [`http_client`](https://docs.redpanda.com/connect/components/inputs/http_client/) input it would look like this: ```yaml input: http_client: url: TODO verb: GET rate_limit: foobar ``` By using a rate limit in this way we can guarantee that our input will only poll our HTTP source at the rate of 500 requests per second. Some components don’t have a `rate_limit` field but we might still wish to throttle them by a rate limit, in which case we can use the [`rate_limit` processor](https://docs.redpanda.com/connect/components/processors/rate_limit/) that applies back pressure to a processing pipeline when the limit is reached. For example, if we wished to limit the consumption of lines of a [`csv` file input](https://docs.redpanda.com/connect/components/inputs/csv/) to a specified rate limit we can do that with the following: ```yaml input: csv: paths: - ./foo.csv processors: - rate_limit: resource: foobar ``` You can find out more about resources [in this document.](https://docs.redpanda.com/connect/configuration/resources/) --- # Page 308: local **URL**: https://docs.redpanda.com/connect/components/rate_limits/local.md --- # local --- title: local latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: rate_limits/local page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rate_limits/local.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/rate_limits/local.adoc description: A simple X every Y rate limit that can be shared across components within a pipeline. It does not support distributed rate limiting across instances. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A simple X every Y rate limit that can be shared across any number of components within a pipeline. It does not support distributed rate limiting across instances. #### Common ```yml rate_limit: label: "" local: count: 1000 interval: 1s ``` #### Advanced ```yml rate_limit: label: "" local: count: 1000 interval: 1s ``` ## [](#fields)Fields ### [](#count)`count` The maximum number of requests to allow for a given period of time. **Type**: `int` **Default**: `1000` ### [](#interval)`interval` The time window to limit requests by. **Type**: `string` **Default**: `1s` --- # Page 309: redis **URL**: https://docs.redpanda.com/connect/components/rate_limits/redis.md --- # redis --- title: redis latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: rate_limits/redis page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rate_limits/redis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/rate_limits/redis.adoc description: A fixed-window rate limit backed by Redis, shared by every Redpanda Connect instance that uses the same Redis instance and key. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- A rate limit implementation using Redis. It works by using a simple token bucket algorithm to limit the number of requests to a given count within a given time period. The rate limit is shared across all instances of Redpanda Connect that use the same Redis instance, which must all have a consistent count and interval. Introduced in version 4.12.0. #### Common ```yml rate_limit: label: "" redis: url: "" # No default (required) count: 1000 interval: 1s key: "" # No default (required) ``` #### Advanced ```yml rate_limit: label: "" redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] count: 1000 interval: 1s key: "" # No default (required) ``` ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#count)`count` The maximum number of requests allowed in each `interval`. Requests are counted in Redis under `key`, so every Redpanda Connect instance that uses the same Redis instance and the same `key` shares this limit. The value must be greater than zero. **Type**: `int` **Default**: `1000` ### [](#interval)`interval` The time window to limit requests by. **Type**: `string` **Default**: `1s` ### [](#key)`key` The key to use for the rate limit. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 310: redpanda **URL**: https://docs.redpanda.com/connect/components/redpanda/about.md --- # redpanda --- title: redpanda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: redpanda/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: redpanda/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/redpanda/about.adoc description: Learn how the redpanda configuration block connects Redpanda Connect to a Redpanda cluster for pipeline logs and status. page-git-created-date: "2024-05-29" page-git-modified-date: "2026-10-14" --- > 📝 **NOTE** > > This component requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. The Redpanda Connect configuration service allows you to: - Configure Redpanda cluster credentials in a single configuration block, which is referenced by multiple components in data pipeline. For more information, see the [Pipeline example](#pipeline-example). - Send logs and status updates to topics on a Redpanda cluster, in addition to the [default logger](https://docs.redpanda.com/connect/components/logger/about/). The `redpanda` namespace contains the configuration of this service. #### Common ```yml redpanda: seed_brokers: [] # No default (required) pipeline_id: "" logs_topic: "" logs_level: info status_topic: "" ``` #### Advanced ```yml redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s pipeline_id: "" logs_topic: "" logs_level: info status_topic: "" partitioner: "" # No default (optional) idempotent_write: true acks: all compression: snappy allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: "0" max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda: topics: [ topic_A, topic_B ] consumer_group: my_group output: redpanda: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#status-events)Status events When you set `status_topic`, Redpanda Connect emits status events to that topic as pipelines start, run, and stop. For monitoring and troubleshooting guidance, see [Monitor Pipeline Status](https://docs.redpanda.com/connect/guides/monitor-pipeline-status/). ### [](#status-event-types)Status event types Status events are emitted during the pipeline lifecycle. All events include `pipeline_id`, `instance_id`, and `timestamp` fields. **TYPE\_INITIALIZING** (value: 1) Emitted when a pipeline instance has successfully parsed its configuration and is attempting to start. This is the first event sent after pipeline startup. Example: ```json { "type": "TYPE_INITIALIZING", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890000 } ``` **TYPE\_CONNECTION\_HEALTHY** (value: 2) Emitted every 30 seconds (heartbeat) when all pipeline connections (inputs and outputs) are active and functioning normally. Example: ```json { "type": "TYPE_CONNECTION_HEALTHY", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890030 } ``` **TYPE\_CONNECTION\_ERROR** (value: 3) Emitted every 30 seconds (during the same heartbeat cycle) when one or more connections are inactive or experiencing errors. The event includes detailed error information for each failing connection: - `path`: The configuration path of the connector (see [field paths](https://docs.redpanda.com/connect/configuration/field_paths/)) - `label`: Optional label assigned to the connector - `message`: The error message describing the connection failure Example: ```json { "type": "TYPE_CONNECTION_ERROR", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890060, "connection_errors": [ { "path": "input.kafka_franz", "label": "primary-input", "message": "kafka: connection refused" } ] } ``` **TYPE\_EXITING** (value: 4) Emitted when a pipeline instance is shutting down, either gracefully or due to an error. If shutdown was caused by an error, the event includes an `exit_error` object with the error message. Example (graceful shutdown): ```json { "type": "TYPE_EXITING", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890090 } ``` Example (error shutdown): ```json { "type": "TYPE_EXITING", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890090, "exit_error": { "message": "failed to process message: invalid format" } } ``` ### [](#message-format)Message format Status events are written to the configured topic as Protocol Buffer JSON messages: - **Topic**: The value specified in `status_topic` - **Key**: The `pipeline_id` value (enables tracking all events for a specific pipeline) - **Value**: JSON-encoded status event - **Encoding**: Protocol Buffer JSON (protojson) For the full protobuf schema specification, see the [status.proto definition](https://github.com/redpanda-data/connect/blob/main/proto/redpanda/api/connect/v1alpha1/status.proto). ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. If set to `false`, the topic must already exist. Requires version 4.58.0 or later. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. Requires version 4.35.0 or later. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. Requires version 4.21.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` The compression codec for produced record batches. A batch that compression does not make smaller is sent uncompressed, whichever codec you choose. **Type**: `string` **Default**: `snappy` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Requires version 4.53.0 or later. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the `IDEMPOTENT_WRITE` permission on the `CLUSTER` resource. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally. Requires version 4.24.0 or later. **Type**: `bool` **Default**: `true` ### [](#logs_level)`logs_level` The lowest level of the process logs sent to `logs_topic`. **Type**: `string` **Default**: `info` **Options**: `debug`, `info`, `warn`, `error` ### [](#logs_topic)`logs_topic` A topic to send process logs to. **Type**: `string` **Default**: `""` ```yaml # Examples: logs_topic: __redpanda.connect.logs ``` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. Requires version 4.92.0 or later. **Type**: `int` **Default**: `10000` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. While `idempotent_write` is enabled (the default) this must be `1`, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set `idempotent_write` to `false`, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s `max_in_flight` field, which counts message batches being written in parallel rather than produce requests on the wire. A value of `1` is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests. Requires version 4.92.0 or later. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum size of a produced record batch in bytes. A `MESSAGE_TOO_LARGE` error is returned if a batch exceeds this limit. This field maps to the `max.message.bytes` Kafka property. Ensure the Redpanda broker’s `kafka_batch_max_bytes` property is at least as large as this value, see [kafka\_batch\_max\_bytes](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_batch_max_bytes). **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#pipeline_id)`pipeline_id` An optional identifier for the pipeline, this will be present in logs and status updates sent to topics. **Type**: `string` **Default**: `""` ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. Requires version 4.53.0 or later. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. Requires version 4.3.0 or later. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. Requires version 4.69.0 or later. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#status_topic)`status_topic` A topic to send status updates to. **Type**: `string` **Default**: `""` ```yaml # Examples: status_topic: __redpanda.connect.status ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. Requires version 4.3.0 or later. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). Requires version 4.3.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 311: Scanners **URL**: https://docs.redpanda.com/connect/components/scanners/about.md --- # Scanners --- title: Scanners latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/about.adoc description: Learn how scanners work in Redpanda Connect. Scanners define how inputs split a raw stream of bytes into logical messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- For such inputs it’s necessary to define a mechanism by which the stream of source bytes can be chopped into smaller logical messages, processed and outputted as a continuous process whilst the stream is being read, as this dramatically reduces the memory usage of Redpanda Connect as a whole and results in a more fluid flow of data. The way in which we define this chopping mechanism is through scanners, configured as a field on each input that requires one. For example, if we wished to consume files line-by-line, which each individual line being processed as a discrete message, we could use the [`lines` scanner](https://docs.redpanda.com/connect/components/scanners/lines/) with our [`file` input](https://docs.redpanda.com/connect/components/inputs/file/): ## Common ```yaml input: file: paths: [ "./*.txt" ] scanner: lines: {} ``` ## Advanced ```yaml # Instead of newlines, use a custom delimiter: input: file: paths: [ "./*.txt" ] scanner: lines: custom_delimiter: "---END---" max_buffer_size: 100_000_000 # 100MB line buffer ``` A scanner is a plugin similar to any other core Redpanda Connect component (inputs, processors, outputs, etc), which means it’s possible to define your own scanners that can be utilized by inputs that need them. --- # Page 312: avro **URL**: https://docs.redpanda.com/connect/components/scanners/avro.md --- # avro --- title: avro latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/avro page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/avro.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/avro.adoc description: Consume a stream of Avro OCF datum. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consume a stream of Avro OCF datum. #### Common ```yml scanner: avro: {} ``` #### Advanced ```yml scanner: avro: raw_json: false max_decompressed_block_bytes: 16777216 ``` ## [](#avro-json-format)Avro JSON format This scanner creates documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when decoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The `"name"` is the type’s name and the `"value"` is the recursively encoded value. For Avro’s named types (record, fixed or enum), the user-specified name is used. For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - The `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can create documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`raw_json`](#raw_json) to `true`. ## [](#metadata)Metadata This scanner emits the following metadata for each message: - The `@avro_schema` field: The canonical Avro schema. - The `@avro_schema_fingerprint` field: The schema ID or fingerprint. ## [](#fields)Fields ### [](#max_decompressed_block_bytes)`max_decompressed_block_bytes` The maximum size, in bytes, that a single compressed Avro OCF block is allowed to expand to when decompressed. This guards against decompression-amplification ("deflate bomb") inputs, where a tiny compressed block would otherwise expand into a very large allocation. A block that exceeds this limit causes the scan to fail rather than be materialized. There is deliberately no way to disable this cap: pipelines with legitimately larger blocks should set an explicit higher value. Set to `0` to use the default (equivalent to omitting the field). Requires version 4.109.0 or later. **Type**: `int` **Default**: `16777216` ### [](#raw_json)`raw_json` Whether to decode messages into normal JSON rather than [Avro JSON](https://avro.apache.org/docs/1.12.0/specification/#json-encoding). When true, this unwraps union values (bare values instead of {"type": value} wrappers). **Type**: `bool` **Default**: `false` --- # Page 313: chunker **URL**: https://docs.redpanda.com/connect/components/scanners/chunker.md --- # chunker --- title: chunker latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/chunker page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/chunker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/chunker.adoc description: Split an input stream into chunks of a given number of bytes. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Split an input stream into chunks of a given number of bytes. #### Common ```yml scanner: chunker: size: 0 # No default (required) ``` #### Advanced ```yml scanner: chunker: size: 0 # No default (required) ``` ## [](#fields)Fields ### [](#size)`size` The size of each chunk in bytes. **Type**: `int` --- # Page 314: csv **URL**: https://docs.redpanda.com/connect/components/scanners/csv.md --- # csv --- title: csv latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/csv page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/csv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/csv.adoc description: Consume comma-separated values row by row, including support for custom delimiters. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consume comma-separated values row by row, including support for custom delimiters. Introduced in version 4.25.0. ## [](#metadata)Metadata This scanner adds the following metadata to each message: - `csv_row` The index of each row, beginning at 0. ### Common ```yml scanner: csv: custom_delimiter: "" # No default (optional) parse_header_row: true lazy_quotes: false continue_on_error: false ``` ### Advanced ```yml scanner: csv: custom_delimiter: "" # No default (optional) parse_header_row: true lazy_quotes: false continue_on_error: false ``` ## [](#fields)Fields ### [](#continue_on_error)`continue_on_error` If a row fails to parse due to any error emit an empty message marked with the error and then continue consuming subsequent rows when possible. This can sometimes be useful in situations where input data contains individual rows which are malformed. However, when a row encounters a parsing error it is impossible to guarantee that following rows are valid, as this indicates that the input data is unreliable and could potentially emit misaligned rows. **Type**: `bool` **Default**: `false` ### [](#custom_delimiter)`custom_delimiter` Use a provided custom delimiter instead of the default comma. **Type**: `string` ### [](#lazy_quotes)`lazy_quotes` If set to `true`, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. **Type**: `bool` **Default**: `false` ### [](#parse_header_row)`parse_header_row` Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, each message will consist of an array of values from the corresponding CSV row. **Type**: `bool` **Default**: `true` --- # Page 315: decompress **URL**: https://docs.redpanda.com/connect/components/scanners/decompress.md --- # decompress --- title: decompress latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/decompress page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/decompress.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/decompress.adoc description: Decompress the stream of bytes according to an algorithm, before feeding it into a child scanner. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Decompress the stream of bytes according to an algorithm, before feeding it into a child scanner. #### Common ```yml scanner: decompress: algorithm: "" # No default (required) into: to_the_end: {} ``` #### Advanced ```yml scanner: decompress: algorithm: "" # No default (required) into: to_the_end: {} ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` One of `gzip`, `pgzip`, `zlib`, `bzip2`, `flate`, `snappy`, `lz4`, `zstd`. **Type**: `string` ### [](#into)`into` The child scanner to feed the decompressed stream into. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 316: json_array **URL**: https://docs.redpanda.com/connect/components/scanners/json_array.md --- # json_array --- title: json_array latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/json_array page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/json_array.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/json_array.adoc description: Consumes a stream of one or more JSON elements within a top level array. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-10-10" --- **Introduced in version 4.65.0.** Consumes a stream of one or more JSON elements within a top level array. This scanner is useful for: - Processing exports from systems that generate a JSON array as the top-level JSON structure (for example, logs, bulk exports, etc). - Efficiently breaking up large files with many objects into individual events/messages. Suppose you have a file `events.json`: `events.json` ```json [ {"event": "login", "user": "alice"}, {"event": "logout", "user": "bob"}, {"event": "purchase", "user": "carol", "amount": 42} ] ``` The configuration to process this file is: ```yaml input: file: paths: [ "./events.json" ] scanner: json_array: {} ``` Result: Each event in the array is processed as a separate message. ## [](#requirements)Requirements The `json_array` scanner expects the input to be a single JSON array, where each array element is a JSON object or value. ## [](#fields)Fields The `json_array` scanner has no required fields. You declare it as `{}` in your config. ```yaml json_array: {} ``` --- # Page 317: json_documents **URL**: https://docs.redpanda.com/connect/components/scanners/json_documents.md --- # json_documents --- title: json_documents latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/json_documents page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/json_documents.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/json_documents.adoc description: Consumes a stream of one or more JSON documents. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consumes a stream of one or more JSON documents. Introduced in version 4.27.0. ## Common ```yml scanner: json_documents: {} ``` ## Advanced ```yml scanner: json_documents: {} ``` --- # Page 318: lines **URL**: https://docs.redpanda.com/connect/components/scanners/lines.md --- # lines --- title: lines latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/lines page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/lines.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/lines.adoc description: Split an input stream into a message per line of data. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Split an input stream into a message per line of data. #### Common ```yml scanner: lines: custom_delimiter: "" # No default (optional) max_buffer_size: 65536 omit_empty: false ``` #### Advanced ```yml scanner: lines: custom_delimiter: "" # No default (optional) max_buffer_size: 65536 omit_empty: false ``` ## [](#fields)Fields ### [](#custom_delimiter)`custom_delimiter` Use a provided custom delimiter for detecting the end of a line rather than a single line break. **Type**: `string` ### [](#max_buffer_size)`max_buffer_size` Set the maximum buffer size for storing line data, this limits the maximum size that a line can be without causing an error. **Type**: `int` **Default**: `65536` ### [](#omit_empty)`omit_empty` Omit empty lines. Requires version 4.30.0 or later. **Type**: `bool` **Default**: `false` --- # Page 319: re_match **URL**: https://docs.redpanda.com/connect/components/scanners/re_match.md --- # re_match --- title: re_match latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/re_match page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/re_match.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/re_match.adoc description: Split an input stream into segments matching against a regular expression. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Split an input stream into segments matching against a regular expression. #### Common ```yml scanner: re_match: pattern: "" # No default (required) max_buffer_size: 65536 ``` #### Advanced ```yml scanner: re_match: pattern: "" # No default (required) max_buffer_size: 65536 ``` ## [](#fields)Fields ### [](#max_buffer_size)`max_buffer_size` Set the maximum buffer size for storing line data, this limits the maximum size that a message can be without causing an error. **Type**: `int` **Default**: `65536` ### [](#pattern)`pattern` The pattern to match against. **Type**: `string` ```yaml # Examples: pattern: (?m)^\d\d:\d\d:\d\d ``` --- # Page 320: skip_bom **URL**: https://docs.redpanda.com/connect/components/scanners/skip_bom.md --- # skip_bom --- title: skip_bom latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/skip_bom page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/skip_bom.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/skip_bom.adoc description: Skip one or more byte order marks for each opened child scanner. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Skip one or more byte order marks for each opened child scanner. #### Common ```yml scanner: skip_bom: into: to_the_end: {} ``` #### Advanced ```yml scanner: skip_bom: into: to_the_end: {} ``` ## [](#fields)Fields ### [](#into)`into` The child scanner to feed the resulting stream into. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 321: switch **URL**: https://docs.redpanda.com/connect/components/scanners/switch.md --- # switch --- title: switch latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/switch page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/switch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/switch.adoc description: Select a child scanner dynamically for source data based on factors such as the filename. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Select a child scanner dynamically for source data based on factors such as the filename. Introduced in version 4.25.0. This scanner outlines a list of potential child scanner candidates to be chosen, and for each source of data the first candidate to pass will be selected. A candidate without any conditions acts as a catch-all and will pass for every source, it is recommended to always have a catch-all scanner at the end of your list. If a given source of data does not pass a candidate an error is returned and the data is rejected. #### Common ```yml scanner: switch: - re_match_name: "" # No default (optional) scanner: {} # No default (required) ``` #### Advanced ```yml scanner: switch: - re_match_name: "" # No default (optional) scanner: {} # No default (required) ``` ## [](#fields)Fields ### [](#re_match_name)`re_match_name` A regular expression to test against the name of each source of data fed into the scanner (filename or equivalent). If this pattern matches the child scanner is selected. **Type**: `string` ### [](#scanner)`scanner` The scanner to activate if this candidate passes. **Type**: `scanner` ## [](#examples)Examples ### [](#switch-based-on-file-name)Switch based on file name In this example a file input chooses a scanner based on the extension of each file ```yaml input: file: paths: [ ./data/* ] scanner: switch: - re_match_name: '\.avro$' scanner: { avro: {} } - re_match_name: '\.csv$' scanner: { csv: {} } - re_match_name: '\.csv.gz$' scanner: decompress: algorithm: gzip into: csv: {} - re_match_name: '\.tar$' scanner: { tar: {} } - re_match_name: '\.tar.gz$' scanner: decompress: algorithm: gzip into: tar: {} - scanner: { to_the_end: {} } ``` --- # Page 322: tar **URL**: https://docs.redpanda.com/connect/components/scanners/tar.md --- # tar --- title: tar latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/tar page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/tar.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/tar.adoc description: Consume a tar archive file by file. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Consume a tar archive file by file. Introduced in version 4.25.0. ## [](#metadata)Metadata This scanner adds the following metadata to each message: - `tar_name` ### Common ```yml scanner: tar: {} ``` ### Advanced ```yml scanner: tar: {} ``` --- # Page 323: to_the_end **URL**: https://docs.redpanda.com/connect/components/scanners/to_the_end.md --- # to_the_end --- title: to_the_end latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: scanners/to_the_end page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/to_the_end.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/to_the_end.adoc description: Read the input stream all the way until the end and deliver it as a single message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Read the input stream all the way until the end and deliver it as a single message. Introduced in version 4.25.0. > ⚠️ **CAUTION** > > Some sources of data may not have a logical end, therefore caution should be made to exclusively use this scanner when the end of an input stream is clearly defined (and well within memory). ## Common ```yml scanner: to_the_end: {} ``` ## Advanced ```yml scanner: to_the_end: {} ``` --- # Page 324: Tracers **URL**: https://docs.redpanda.com/connect/components/tracers/about.md --- # Tracers --- title: Tracers latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tracers/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/about.adoc description: Learn how tracers work in Redpanda Connect. A tracer sends distributed tracing events to a destination such as Jaeger. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- A tracer type represents a destination for Redpanda Connect to send tracing events to such as [Jaeger](https://www.jaegertracing.io/). When a tracer is configured all messages will be allocated a root span during ingestion that represents their journey through a Redpanda Connect pipeline. Many Redpanda Connect processors create spans, and so tracing is a great way to analyse the pathways of individual messages as they progress through a Redpanda Connect instance. Some inputs, such as `http_server` and `http_client`, are capable of extracting a root span from the source of the message (HTTP headers). This is a work in progress and should eventually expand so that all inputs have a way of doing so. Other inputs, such as `kafka` can be configured to extract a root span by using the `extract_tracing_map` field. A tracer config section looks like this: ```yaml tracer: jaeger: agent_address: localhost:6831 sampler_type: const sampler_param: 1 ``` > ⚠️ **CAUTION** > > Although the configuration spec of this component is stable the format of spans, tags and logs created by Redpanda Connect is subject to change as it is tuned for improvement. --- # Page 325: gcp_cloudtrace **URL**: https://docs.redpanda.com/connect/components/tracers/gcp_cloudtrace.md --- # gcp_cloudtrace --- title: gcp_cloudtrace latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tracers/gcp_cloudtrace page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/gcp_cloudtrace.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/gcp_cloudtrace.adoc description: Send tracing events to Google Cloud Trace. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Send tracing events to a [Google Cloud Trace](https://cloud.google.com/trace). Introduced in version 4.2.0. #### Common ```yml tracer: gcp_cloudtrace: project: "" # No default (required) sampling_ratio: 1 flush_interval: "" # No default (optional) ``` #### Advanced ```yml tracer: gcp_cloudtrace: project: "" # No default (required) sampling_ratio: 1 tags: {} flush_interval: "" # No default (optional) ``` ## [](#fields)Fields ### [](#flush_interval)`flush_interval` The maximum time to wait before exporting a batch of tracing spans. When unset, the `OTEL_BSP_SCHEDULE_DELAY` environment variable is used if set, otherwise the OpenTelemetry SDK default of `5s`. **Type**: `string` ### [](#project)`project` The google project with Cloud Trace API enabled. If this is omitted then the Google Cloud SDK will attempt auto-detect it from the environment. **Type**: `string` ### [](#sampling_ratio)`sampling_ratio` Sets the ratio of traces to sample. Tuning the sampling ratio is recommended for high-volume production workloads. **Type**: `float` **Default**: `1` ```yaml # Examples: sampling_ratio: 1 ``` ### [](#tags)`tags` A map of tags to add to tracing spans. **Type**: `object` **Default**: `{}` --- # Page 326: jaeger **URL**: https://docs.redpanda.com/connect/components/tracers/jaeger.md --- # jaeger --- title: jaeger latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tracers/jaeger page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/jaeger.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/jaeger.adoc description: Send tracing events to a Jaeger agent or collector. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Send tracing events to a [Jaeger](https://www.jaegertracing.io/) agent or collector. #### Common ```yml tracer: jaeger: agent_address: "" collector_url: "" sampler_type: const flush_interval: "" # No default (optional) ``` #### Advanced ```yml tracer: jaeger: agent_address: "" collector_url: "" sampler_type: const sampler_param: 1 tags: {} flush_interval: "" # No default (optional) ``` ## [](#fields)Fields ### [](#agent_address)`agent_address` The address of a Jaeger agent to send tracing events to. **Type**: `string` **Default**: `""` ```yaml # Examples: agent_address: jaeger-agent:6831 ``` ### [](#collector_url)`collector_url` The URL of a Jaeger collector to send tracing events to. If set, this will override `agent_address`. Requires version 3.38.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: collector_url: https://jaeger-collector:14268/api/traces ``` ### [](#flush_interval)`flush_interval` The maximum time to wait before exporting a batch of tracing spans. When unset, the `OTEL_BSP_SCHEDULE_DELAY` environment variable is used if set, otherwise the OpenTelemetry SDK default of `5s`. **Type**: `string` ### [](#sampler_param)`sampler_param` A parameter to use for sampling. This field is unused for some sampling types. **Type**: `float` **Default**: `1` ### [](#sampler_type)`sampler_type` The sampler type to use. **Type**: `string` **Default**: `const` | Option | Summary | | --- | --- | | const | Sample a percentage of traces. 1 or more means all traces are sampled, 0 means no traces are sampled and anything in between means a percentage of traces are sampled. Tuning the sampling rate is recommended for high-volume production workloads. | ### [](#tags)`tags` A map of tags to add to tracing spans. **Type**: `object` **Default**: `{}` --- # Page 327: none **URL**: https://docs.redpanda.com/connect/components/tracers/none.md --- # none --- title: none latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tracers/none page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/none.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/none.adoc description: Do not send tracing events anywhere. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Do not send tracing events anywhere. ## Common ```yml tracer: none: {} ``` ## Advanced ```yml tracer: none: {} ``` --- # Page 328: open_telemetry_collector **URL**: https://docs.redpanda.com/connect/components/tracers/open_telemetry_collector.md --- # open_telemetry_collector --- title: open_telemetry_collector latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tracers/open_telemetry_collector page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/open_telemetry_collector.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/open_telemetry_collector.adoc description: Send tracing events to an Open Telemetry collector. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Send tracing events to an [Open Telemetry collector](https://opentelemetry.io/docs/collector/). #### Common ```yml tracer: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) sampling: enabled: false ratio: 0 # No default (optional) ``` #### Advanced ```yml tracer: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) tags: {} sampling: enabled: false ratio: 0 # No default (optional) ``` ## [](#fields)Fields ### [](#grpc)`grpc[]` A list of grpc collectors. **Type**: `array` ### [](#grpc-address)`grpc[].address` The endpoint of a collector to send events to. Requires version 4.26.0 or later. **Type**: `string` ```yaml # Examples: address: localhost:4317 ``` ### [](#grpc-secure)`grpc[].secure` Connect to the collector with client transport security Requires version 4.20.0 or later. **Type**: `bool` **Default**: `false` ### [](#http)`http[]` A list of http collectors. **Type**: `array` ### [](#http-address)`http[].address` The endpoint of a collector to send events to. Requires version 4.26.0 or later. **Type**: `string` ```yaml # Examples: address: localhost:4318 ``` ### [](#http-secure)`http[].secure` Connect to the collector over HTTPS Requires version 4.20.0 or later. **Type**: `bool` **Default**: `false` ### [](#sampling)`sampling` Settings for trace sampling. Sampling is recommended for high-volume production workloads. Requires version 4.25.0 or later. **Type**: `object` ### [](#sampling-enabled)`sampling.enabled` Whether to enable sampling. **Type**: `bool` **Default**: `false` ### [](#sampling-ratio)`sampling.ratio` Sets the ratio of traces to sample. **Type**: `float` ```yaml # Examples: ratio: 0.85 # --- ratio: 0.5 ``` ### [](#service)`service` The name of the service in traces. Requires version 4.54.0 or later. **Type**: `string` **Default**: `benthos` ### [](#tags)`tags` A map of tags to add to all exported spans and metrics. **Type**: `object` **Default**: `{}` --- # Page 329: redpanda **URL**: https://docs.redpanda.com/connect/components/tracers/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tracers/redpanda page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/redpanda.adoc description: Send tracing events to a Redpanda topic. page-git-created-date: "2025-12-02" page-git-modified-date: "2026-10-10" --- Export distributed tracing data to a Redpanda topic, enabling you to monitor and debug your Redpanda Connect pipelines. Traces are exported in OpenTelemetry format as JSON, allowing integration with observability platforms like Jaeger, Grafana Tempo, or custom trace consumers. #### Common ```yml tracer: redpanda: seed_brokers: [] # No default (required) topic: otel-traces format: json schema_registry: url: "" # No default (optional) service: redpanda-connect sampling: enabled: false ratio: 0 # No default (optional) ``` #### Advanced ```yml tracer: redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s partitioner: "" # No default (optional) idempotent_write: true acks: all compression: snappy allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: "0" max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s topic: otel-traces format: json schema_registry: url: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} service: redpanda-connect tags: {} sampling: enabled: false ratio: 0 # No default (optional) ``` This tracer automatically captures trace spans as messages flow through your pipeline, recording timing information, component metadata, and error details. Use this to: - **Track message flow** through complex pipelines with multiple processors. - **Identify performance bottlenecks** by analyzing span durations. - **Debug failures** by examining trace context and error details. - **Monitor pipeline health** across distributed Redpanda Connect instances. - **Correlate activity** across multiple services using trace IDs. The tracer writes to a dedicated Redpanda topic that can be consumed by trace analysis tools. Configure sampling to control trace volume in high-throughput environments. ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. If set to `false`, the topic must already exist. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` The compression codec for produced record batches. A batch that compression does not make smaller is sent uncompressed, whichever codec you choose. **Type**: `string` **Default**: `snappy` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#format)`format` The serialization format for individual spans in the topic. **Type**: `string` **Default**: `json` | Option | Summary | | --- | --- | | json | Emit in JSON Format | | protobuf | Emit in Protobuf Format | | schema-registry-json | Emit in JSON Format with Schema Registry encoding | | schema-registry-protobuf | Emit in Protobuf Format with Schema Registry encoding | ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the `IDEMPOTENT_WRITE` permission on the `CLUSTER` resource. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally. **Type**: `bool` **Default**: `true` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. Requires version 4.92.0 or later. **Type**: `int` **Default**: `10000` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. While `idempotent_write` is enabled (the default) this must be `1`, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set `idempotent_write` to `false`, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s `max_in_flight` field, which counts message batches being written in parallel rather than produce requests on the wire. A value of `1` is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests. Requires version 4.92.0 or later. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum size of a produced record batch in bytes. A `MESSAGE_TOO_LARGE` error is returned if a batch exceeds this limit. This field maps to the `max.message.bytes` Kafka property. Ensure the Redpanda broker’s `kafka_batch_max_bytes` property is at least as large as this value, see [kafka\_batch\_max\_bytes](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_batch_max_bytes). **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. Requires version 4.92.0 or later. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts. This field is roughly equivalent to Apache Kafka’s `request.timeout.ms` parameter, but grants extra time to requests that have timeout fields. **Type**: `string` **Default**: `10s` ### [](#sampling)`sampling` Configure trace sampling to control the volume of trace data. Sampling is recommended for high-volume production workloads to prevent trace data from overwhelming your observability infrastructure. **Type**: `object` ### [](#sampling-enabled)`sampling.enabled` Whether to enable trace sampling. When disabled, all traces are exported. When enabled, traces are sampled according to the configured ratio. **Type**: `bool` **Default**: `false` ### [](#sampling-ratio)`sampling.ratio` The sampling ratio as a decimal between 0 and 1. For example, `0.1` samples 10% of traces, `0.01` samples 1%. Lower ratios reduce trace volume and overhead. For high-throughput production systems, start with 0.01-0.1 and adjust based on your needs. **Type**: `float` ```yaml # Examples: ratio: 0.05 # --- ratio: 0.85 # --- ratio: 0.5 ``` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when `sasl.mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region in which your resources are hosted. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Schema registry information to publish schemas for tracing data along with the data. Requires version 4.73.0 or later. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Configures authentication with OAuth version 2 using the client credentials token flow. To exchange a refresh token instead, add a `grant_type` parameter with the value `refresh_token` and a `refresh_token` parameter with the token to `endpoint_params`. Requires version 4.74.0 or later. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A map of optional parameters to send to the token provider with client credentials token requests. Each value must be an array of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#service)`service` The service name to identify this Redpanda Connect instance in traces. This appears in trace visualizations and helps correlate traces across distributed systems. Use descriptive names like `order-processor` or `analytics-pipeline`. **Type**: `string` **Default**: `redpanda-connect` ### [](#tags)`tags` Custom key-value tags to attach to all traces from this instance. Use tags to add metadata like environment (`production`, `staging`), region, version, or instance identifiers. Tags appear as resource attributes in OpenTelemetry traces. **Type**: `object` **Default**: `{}` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **Unresponsive hosts**: Set `connect_timeout` to limit how long a connection attempt can take (the default `0s` sets no limit) - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The Redpanda topic where trace data is written. This topic should be dedicated to traces and configured with appropriate retention policies. Default: `otel-traces` **Type**: `string` **Default**: `otel-traces` ## [](#examples)Examples ### [](#basic-tracing-setup)Basic tracing setup Enable tracing for a pipeline, sending all trace data to a local Redpanda cluster. ```yaml input: redpanda: seed_brokers: [ "localhost:9092" ] topics: [ "orders" ] consumer_group: order-processor pipeline: processors: - mapping: | root = this root.processed_at = now() output: redpanda: seed_brokers: [ "localhost:9092" ] topic: processed_orders tracer: redpanda: seed_brokers: [ "localhost:9092" ] topic: otel-traces service: order-processor ``` ### [](#production-tracing-with-sampling)Production tracing with sampling Configure tracing for a production environment with sampling to reduce trace volume, and with TLS and SASL authentication. ```yaml input: redpanda: seed_brokers: [ "redpanda-prod:9092" ] topics: [ "events" ] consumer_group: event-processor pipeline: processors: - branch: request_map: 'root = this' processors: - mapping: 'root.enriched = true' output: redpanda: seed_brokers: [ "redpanda-prod:9092" ] topic: processed_events tracer: redpanda: seed_brokers: [ "redpanda-prod:9092" ] topic: otel-traces service: event-processor-prod sampling: enabled: true ratio: 0.1 # Sample 10% of traces tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 username: tracer-user password: ${TRACER_PASSWORD} ``` ### [](#multi-instance-tracing)Multi-instance tracing Configure tracing for multiple Redpanda Connect instances with unique service names for correlation. Each instance writes to the same topic and sets its own `service` name and `instance` tag. This configuration is for an ingestion service. A processing service uses the same configuration with `service: processing-service` and `instance: processing-01`. ```yaml tracer: redpanda: seed_brokers: [ "redpanda:9092" ] topic: otel-traces service: ingestion-service tags: environment: production region: us-west-2 instance: ingestion-01 ``` --- # Page 330: Configuration **URL**: https://docs.redpanda.com/connect/configuration/about.md --- # Configuration --- title: Configuration latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/about.adoc description: Learn about different options for configuring Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-05-26" --- Redpanda Connect pipelines are configured in a YAML file that consists of a number of root sections, arranged like so: #### Common ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` #### Full ```yaml http: address: 0.0.0.0:4195 debug_endpoints: false input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: none: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' input_resources: [] cache_resources: [] processor_resources: [] rate_limit_resources: [] output_resources: [] logger: level: INFO static_fields: '@service': benthos metrics: prometheus: {} tracer: none: {} shutdown_timeout: 20s shutdown_delay: "" ``` Most sections represent a component type, which you can read about in more detail in [this document](https://docs.redpanda.com/connect/components/about/). These types are hierarchical. For example, an `input` can have a list of child `processor` types attached to it, which in turn can have their own `processor` children. This is powerful but can potentially lead to large and cumbersome configuration files. This document outlines tooling provided by Redpanda Connect to help with writing and managing these more complex configuration files. ## [](#testing)Testing For guidance on how to write and run unit tests for your configuration files read [this guide](https://docs.redpanda.com/connect/configuration/unit_testing/). ## [](#customizing-your-configuration)Customizing your configuration Sometimes it’s useful to write a configuration where certain fields can be defined during deployment. For this purpose Redpanda Connect supports [environment variable interpolation](https://docs.redpanda.com/connect/configuration/interpolation/), allowing you to set fields in your config with environment variables like so: ```yaml input: kafka: addresses: - ${KAFKA_BROKER:localhost:9092} topics: - ${KAFKA_TOPIC:default-topic} ``` This is very useful for sharing configuration files across different deployment environments. ## [](#labels)Labels Labels are unique, user-defined identifiers used throughout Redpanda Connect configurations. They serve two purposes: - **Reference:** Allow different parts of your pipeline to refer to specific components or resources. - **Readability:** Make your configuration more understandable for humans, especially in complex deployments. You can assign labels to most pipeline components, including resources, inputs, outputs, processors, and entire pipelines. Using clear, descriptive labels improves both maintainability and clarity. Labels are commonly applied to the following components: ### [](#resources)Resources Labels identify [reusable resources](#reuse) such as processors, caches, and rate limiters, making them easy to reference elsewhere in your pipeline. ```yaml processor_resources: - label: my-transformer # Processor resource label mapping: 'root = content().uppercase()' cache_resources: - label: user-cache # Cache resource label memory: default_ttl: 300s rate_limit_resources: - label: api-limiter # Rate limiter resource label local: count: 100 interval: 1m ``` ### [](#pipelines-streams-mode)Pipelines (streams mode) When running in [streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/) with the Streams API, labels serve as pipeline names. This enables you to create and reference pipelines by their label through the API. ```bash # Create a pipeline labeled "data-processor" curl -X POST http://localhost:4195/streams/data-processor \ -H "Content-Type: application/yaml" \ -d @pipeline-config.yaml # Reference a pipeline by its label curl http://localhost:4195/streams/data-processor ``` ### [](#component-labeling-for-clarity)Component labeling for clarity You can also use labels on inputs, outputs, processors, and other components to improve the human-readability of your configuration and make troubleshooting easier. For example: ```yaml input: label: ingest_api http_server: {} pipeline: label: user_data_ingest processors: - label: sanitize_fields mapping: 'root = this.trim()' - resource: my-transformer ``` ## [](#label-naming-requirements)Label naming requirements Labels must meet the following criteria: - **Length**: 3-128 characters - **Allowed characters**: Alphanumeric, hyphens, and underscores (`A-Za-z0-9-_`) - **Case sensitivity**: Labels are case-sensitive Example valid labels my-processor data\_transformer\_01 UserAnalytics-v2 Example invalid labels ab // Too short (less than 3 characters) my.processor // Invalid character: period my processor // Invalid character: space ## [](#reuse)Reusing configuration snippets Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component [as a resource](https://docs.redpanda.com/connect/configuration/resources/). In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to "cleanse" the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource `get_foo`: ```yaml pipeline: processors: - resource: get_foo - catch: - mapping: | root = this root.content = this.content.strip_html() - resource: get_foo processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` ### [](#feature-toggles)Feature toggles Resources can be imported separately to your config file with the cli flag `-r` or `-resources`, which is a useful way to switch out resources with common names based on your chosen environment. For example, with a main configuration file `config.yaml`: ```yaml pipeline: processors: - resource: get_foo ``` And then two resource files, one stored at the path `./staging/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` And another stored at the path `./production/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` We can select our chosen resource by changing which file we import, either running: ```bash rpk connect run -r ./staging/request.yaml ./config.yaml ``` Or: ```bash rpk connect run -r ./production/request.yaml ./config.yaml ``` These flags also support wildcards, which allows you to import an entire directory of resource files like `rpk connect run -r "./staging/*.yaml" ./config.yaml`. You can find out more about configuration resources in the [resources document](https://docs.redpanda.com/connect/configuration/resources/). ### [](#templating)Templating Resources can only be instantiated with a single configuration, which means they aren’t suitable for cases where the configuration is required in multiple places but with slightly different parameters. Redpanda Connect has a (currently experimental) alternative feature called templates, with which it’s possible to define a custom configuration schema and a template for building a configuration from that schema. You can read more about templates [in this guide](https://docs.redpanda.com/connect/configuration/templating/). ## [](#reloading)Reloading It’s possible to have a running instance of Redpanda Connect reload configurations, including resource files imported with `-r`/`--resources`, automatically when the files are updated without needing to manually restart the service. This is done by specifying the `-w`/`--watcher` flag when running Redpanda Connect in normal mode or in streams mode: ```bash # Normal mode rpk connect run -w -r ./production/request.yaml ./config.yaml ``` ```bash # Streams mode rpk connect streams -w -r ./production/request.yaml ./stream_configs/*.yaml ``` If a file update results in configuration parsing or linting errors then the change is ignored (with logs informing you of the problem) and the previous configuration will continue to be run (until the issues are fixed). ## [](#enabling-discovery)Enabling discovery The discoverability of configuration fields is a common headache with any configuration driven application. The classic solution is to provide curated documentation that is often hosted on a dedicated site. However, a user often only needs to get their hands on a short, runnable example config file for their use case. They just need to see the format and field names as the fields themselves are usually self explanatory. Forcing such a user to navigate a website, scrolling through paragraphs of text, seems inefficient when all they actually needed to see was something like: ```yaml input: amqp_0_9: urls: [ amqp://guest:guest@localhost:5672/ ] consumer_tag: benthos-consumer queue: benthos-queue prefetch_count: 10 prefetch_size: 0 output: stdout: {} ``` In order to make this process easier Redpanda Connect is able to generate usable configuration examples for any types, and you can do this from the binary using the `create` subcommand. If, for example, we wanted to generate a config with a websocket input, a Kafka output and a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) in the middle, we could do it with the following command: ```bash rpk connect create websocket/mapping/kafka ``` > 💡 **TIP** > > To see which components Redpanda Connect offers, use `rpk connect list`. All of these generated configuration examples also include other useful config sections such as `metrics`, `logging`, etc with sensible defaults. For more information read the output from `rpk connect create --help`. ## [](#help-with-debugging)Help with debugging Once you have a config written you now move onto the next headache of proving that it works, and understanding why it doesn’t. Redpanda Connect, like most good config driven services, performs validation on configs and tries to provide sensible error messages. However, with validation it can be hard to capture all problems, and the user usually understands their intentions better than the service. In order to help expose and diagnose config errors Redpanda Connect provides two mechanisms, linting and echoing. ### [](#linting)Linting If you attempt to run a config that has linting errors Redpanda Connect will print the errors and halt execution. If, however, you want to test your configs before deployment you can do so with the `lint` subcommand: For example, imagine we have a config `foo.yaml`, where we intend to read from AMQP, but there is a typo in our config struct: ```text input: amqp_0_9: yourl: amqp://guest:guest@rabbitmqserver:5672/ ``` We can catch this error before attempting to run the config: ```bash rpk connect lint ./foo.yaml ./foo.yaml: line 3: field yourl not recognized ``` For more information read the output from `rpk connect lint --help`. ### [](#echoing)Echoing Echoing is where Redpanda Connect can print back your configuration _after_ it has been parsed. It is done with the `echo` subcommand, which is able to show you a normalized version of your config, allowing you to see how it was interpreted: ```bash rpk connect echo ./your-config.yaml ``` You can check the output of the above command to see if certain sections are missing or fields are incorrect, which allows you to pinpoint typos in the config. ## [](#shutting-down)Shutting down Under normal operating conditions, the Redpanda Connect process will shut down when there are no more messages produced by inputs and the final message has been processed. The shutdown procedure can also be initiated by sending the process a interrupt (`SIGINT`) or termination (`SIGTERM`) signal. There are two top-level configuration options that control the shutdown behavior: `shutdown_timeout` and `shutdown_delay`. ### [](#shutdown-delay)Shutdown delay The `shutdown_delay` option can be used to delay the start of the shutdown procedure. This is useful for pipelines that need a short grace period to have their metrics and traces scraped. While the shutdown delay is in effect, the HTTP metrics endpoint continues to be available for scraping and any active tracers are free to flush remaining traces. The shutdown delay can be interrupted by sending the Redpanda Connect process a second OS interrupt or termination signal. ### [](#shutdown-timeout)Shutdown timeout The `shutdown_timeout` option sets a hard deadline for Redpanda Connect process to gracefully terminate. If this duration is exceeded then the process is forcefully terminated and any messages that were in-flight will be dropped. This option takes effect after the `shutdown_delay` duration has passed if that is enabled. --- # Page 331: Configure an Allow or Deny List **URL**: https://docs.redpanda.com/connect/configuration/allow_and_deny_lists.md --- # Configure an Allow or Deny List --- title: Configure an Allow or Deny List latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: allow_and_deny_lists page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: allow_and_deny_lists.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/allow_and_deny_lists.adoc description: Learn how to configure an allow or deny list for a Redpanda Connect instance. page-git-created-date: "2024-11-19" page-git-modified-date: "2025-08-08" --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. You can configure an allow or deny list to limit the Redpanda Connect components that users can run within data pipelines on a Redpanda Connect instance. Introduced in version 4.39.0. ## [](#limitations)Limitations - You can configure only one allow list, or one deny list per Redpanda Connect instance. - When you add the name of a component to your allow or deny list, you grant or block access to all related components. For example, if you include `aws_s3` in your list, you grant or block access to the entire `aws_s3` family of components, including the cache, input, and output. ## [](#configure-an-allow-list)Configure an allow list To configure an allow list: 1. Create a `connector_list.yaml` file in the following location: `/etc/redpanda/connector_list.yaml`. 2. In the [Connector Catalog](https://docs.redpanda.com/connect/components/about/), find the exact name of the component family you want to grant access to. For example: `aws_s3` and `azure_cosmosdb`. 3. Add the names of the component families to the `connector_list.yaml` file, using the following format: ```yaml allow: - aws_s3 - azure_cosmosdb ``` 4. Save your configuration. ## [](#configure-a-deny-list)Configure a deny list To configure a deny list: 1. Create a `connector_list.yaml` file in the following location: `/etc/redpanda/connector_list.yaml`. 2. In the [Connector Catalog](https://docs.redpanda.com/connect/components/about/), find the exact name of the component family you want to block access to. For example: `aws_s3` and `azure_cosmosdb`. 3. Add the names of the component families to the `connector_list.yaml` file, using the following format: ```yaml deny: - aws_s3 - azure_cosmosdb ``` 4. Save your configuration. --- # Page 332: Message Batching **URL**: https://docs.redpanda.com/connect/configuration/batching.md --- # Message Batching --- title: Message Batching latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: batching page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: batching.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/batching.adoc description: Learn how message batching works in Redpanda Connect, including batch policies and how batches flow between inputs, processors, and outputs. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Redpanda Connect is able to join sources and sinks with sometimes conflicting batching behaviors without sacrificing its strong delivery guarantees. It’s also able to perform powerful [processing functions](https://docs.redpanda.com/connect/configuration/windowed_processing/) across batches of messages such as grouping, archiving and reduction. Therefore, batching within Redpanda Connect is a mechanism that serves multiple purposes: 1. [Performance (throughput)](#performance) 2. [Grouped message processing](#grouped-message-processing) 3. [Compatibility (mixing multi and single part message protocols)](#compatibility) ## [](#performance)Performance For most users the only benefit of batching messages is improving throughput over your output protocol. For some protocols this can happen in the background and requires no configuration from you. However, if an output has a `batching` configuration block this means it benefits from batching and requires you to specify how you’d like your batches to be formed by configuring a [batching policy](#batch-policy): ```yaml output: kafka: addresses: [ todo:9092 ] topic: benthos_stream # Either send batches when they reach 10 messages or when 100ms has passed # since the last batch. batching: count: 10 period: 100ms ``` However, a small number of inputs such as [`kafka`](https://docs.redpanda.com/connect/components/inputs/kafka/) must be consumed sequentially (in this case by partition) and therefore benefit from specifying your batch policy at the input level instead: ```yaml input: kafka: addresses: [ todo:9092 ] topics: [ benthos_input_stream ] batching: count: 10 period: 100ms output: kafka: addresses: [ todo:9092 ] topic: benthos_stream ``` Inputs that behave this way are documented as such and have a `batching` configuration block. Sometimes you may prefer to create your batches before processing in order to benefit from [batch wide processing](#grouped-message-processing), in which case if your input doesn’t already support [a batch policy](#batch-policy) you can instead use a [`broker`](https://docs.redpanda.com/connect/components/inputs/broker/), which also allows you to combine inputs with a single batch policy: ```yaml input: broker: inputs: - resource: foo - resource: bar batching: count: 50 period: 500ms ``` This also works the same with [output brokers](https://docs.redpanda.com/connect/components/outputs/broker/). ## [](#grouped-message-processing)Grouped message processing And some processors such as [`while`](https://docs.redpanda.com/connect/components/processors/while/) are executed once across a whole batch, you can avoid this behavior with the [`for_each` processor](https://docs.redpanda.com/connect/components/processors/for_each/): ```yaml pipeline: processors: - for_each: - while: at_least_once: true max_loops: 0 check: errored() processors: - catch: [] # Wipe any previous error - resource: foo # Attempt this processor until success ``` There’s a vast number of processors that specialise in operations across batches such as [grouping](https://docs.redpanda.com/connect/components/processors/group_by/) and [archiving](https://docs.redpanda.com/connect/components/processors/archive/). For example, the following processors group a batch of messages according to a metadata field and compresses them into separate `.tar.gz` archives: ```yaml pipeline: processors: - group_by_value: value: ${! meta("kafka_partition") } - archive: format: tar - compress: algorithm: gzip output: aws_s3: bucket: TODO path: docs/${! meta("kafka_partition") }/${! count("files") }-${! timestamp_unix_nano() }.tar.gz ``` For more examples of batched (or windowed) processing check out [this document](https://docs.redpanda.com/connect/configuration/windowed_processing/). ## [](#compatibility)Compatibility Redpanda Connect is able to read and write over protocols that support multiple part messages, and all payloads travelling through Redpanda Connect are represented as a multiple part message. Therefore, all components within Redpanda Connect are able to work with multiple parts in a message as standard. When messages reach an output that _doesn’t_ support multiple parts the message is broken down into an individual message per part, and then one of two behaviors happen depending on the output. If the output supports batch sending messages then the collection of messages are sent as a single batch. Otherwise, Redpanda Connect falls back to sending the messages sequentially in multiple, individual requests. This behavior means that not only can multiple part message protocols be easily matched with single part protocols, but also the concept of multiple part messages and message batches are interchangeable within Redpanda Connect. ### [](#shrinking-batches)Shrinking batches A message batch (or multiple part message) can be broken down into smaller batches using the [`split`](https://docs.redpanda.com/connect/components/processors/split/) processor: ```yaml input: # Consume messages that arrive in three parts. resource: foo processors: # Drop the third part - select_parts: parts: [ 0, 1 ] # Then break our message parts into individual messages - split: size: 1 ``` This is also useful when your input source creates batches that are too large for your output protocol: ```yaml input: aws_s3: bucket: todo pipeline: processors: - decompress: algorithm: gzip - unarchive: format: tar # Limit batch sizes to 5MB - split: byte_size: 5_000_000 ``` ## [](#batch-policy)Batch policy When an input or output component has a config field `batching` that means it supports a batch policy. This is a mechanism that allows you to configure exactly how your batching should work on messages before they are routed to the input or output it’s associated with. Batches are considered complete and will be flushed downstream when either of the following conditions are met: - The `byte_size` field is non-zero and the total size of the batch in bytes matches or exceeds it (disregarding metadata.) - The `count` field is non-zero and the total number of messages in the batch matches or exceeds it. - A message added to the batch causes the [`check`](https://docs.redpanda.com/connect/guides/bloblang/about/) to return to `true`. - The `period` field is non-empty and the time since the last batch exceeds its value. This allows you to combine conditions: ```yaml output: kafka: addresses: [ todo:9092 ] topic: benthos_stream # Either send batches when they reach 10 messages or when 100ms has passed # since the last batch. batching: count: 10 period: 100ms ``` > ⚠️ **CAUTION** > > A batch policy has the capability to _create_ batches, but not to break them down. If your configured pipeline is processing messages that are batched _before_ they reach the batch policy then they may circumvent the conditions you’ve specified here, resulting in sizes you aren’t expecting. If you are affected by this limitation then consider breaking the batches down with a [`split` processor](https://docs.redpanda.com/connect/components/processors/split/) before they reach the batch policy. ### [](#post-batch-processing)Post-batch processing A batch policy also has a field `processors` which allows you to define an optional list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to each batch before it is flushed. This is a good place to aggregate or archive the batch into a compatible format for an output: ```yaml output: http_client: url: http://localhost:4195/post batching: count: 10 processors: - archive: format: lines ``` The above config will batch up messages and then merge them into a line delimited format before sending it over HTTP. This is an easier format to parse than the default which would have been [rfc1342](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). During shutdown any remaining messages waiting for a batch to complete will be flushed down the pipeline. --- # Page 333: Dynamic Inputs and Outputs **URL**: https://docs.redpanda.com/connect/configuration/dynamic_inputs_and_outputs.md --- # Dynamic Inputs and Outputs --- title: Dynamic Inputs and Outputs latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: dynamic_inputs_and_outputs page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: dynamic_inputs_and_outputs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/dynamic_inputs_and_outputs.adoc description: Learn how to add, update, and remove inputs and outputs at runtime with the dynamic fan-in and fan-out components and their REST APIs. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- It is possible to have sets of inputs and outputs in Redpanda Connect that can be added, updated and removed during runtime with the [dynamic fan in](https://docs.redpanda.com/connect/components/inputs/dynamic/) and [dynamic fan out](https://docs.redpanda.com/connect/components/outputs/dynamic/) types. Dynamic inputs and outputs are each identified by unique string labels, which are specified when adding them either in configuration or via the HTTP API. The labels are useful when querying which types are active. ## [](#api)API The API for dynamic types (both inputs and outputs) is a collection of HTTP REST endpoints: ### [](#inputs)`/inputs` Returns a JSON object that maps input labels to an object containing details about the input, including uptime and configuration. If the input has terminated naturally the uptime will be set to `stopped`. ```json { "": { "uptime": "", "config": }, ... } ``` ### [](#inputsinput_label)`/inputs/{input_label}` GET returns the configuration of the input idenfified by `input_label`. POST sets the input `input_label` to the body of the request parsed as a JSON configuration. If the input label already exists the previous input is first stopped and removed. DELETE stops and removes the input identified by `input_label`. ### [](#outputs)`/outputs` Returns a JSON object that maps output labels to an object containing details about the output, including uptime and configuration. If the output has terminated naturally the uptime will be set to `stopped`. ```json { "": { "uptime": "", "config": }, ... } ``` ### [](#outputsoutput_label)`/outputs/{output_label}` GET returns the configuration of the output idenfified by `output_label`. POST sets the output `output_label` to the body of the request parsed as a JSON configuration. If the output label already exists the previous output is first stopped and removed. DELETE stops and removes the output identified by `output_label`. A custom prefix can be set for these endpoints in configuration. ## [](#applications)Applications Dynamic types are useful when a platforms data streams might need to change regularly and automatically. It is also useful for triggering batches of platform data, e.g. a cron job can be created to send hourly curl requests that adds a dynamic input to read a file of sample data: ```sh curl http://localhost:4195/inputs/read_sample -d @- << EOF { "file": { "path": "/tmp/line_delim_sample_data.txt" } } EOF ``` Some inputs have a finite lifetime, e.g. `s3` without an SQS queue configured will close once the whole bucket has been read. When a dynamic types lifetime ends the `uptime` field of an input listing will be set to `stopped`. You can use this to write tools that trigger new inputs (to move onto the next bucket, for example). --- # Page 334: Error Handling **URL**: https://docs.redpanda.com/connect/configuration/error_handling.md --- # Error Handling --- title: Error Handling latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: error_handling page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: error_handling.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/error_handling.adoc description: Learn how to handle processing errors in Redpanda Connect, including error checks, retries, and dead-letter queue patterns. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Redpanda Connect supports a range of [processors](https://docs.redpanda.com/connect/components/processors/about/), such as `http` and `aws_lambda`, that may fail when retry attempts are exhausted. By default, when a processor fails, the message data continues through the pipeline mostly unchanged, except for the addition of a metadata flag, which you can use for handling errors. To make processing errors terminal instead, see [Strict error handling](#strict-error-handling). This topic explains some common error-handling patterns, including dropping messages, recovering them with more processing, and routing them to a dead-letter queue. It also shows how to combine these approaches, where appropriate, and provides a method for [trapping unhandled errors](#logging-fatal-errors). ## [](#strict-error-handling)Strict error handling To make processing errors terminal instead of relying on error flags, enable strict error handling with the top-level `error_handling` configuration block: ```yaml error_handling: strict: true ``` When `error_handling.strict` is enabled: - A processing error is terminal for the affected message. The message skips the remaining processors in its pipeline. - The failed message is rejected (nacked) at the output rather than written. - A standalone `catch` processor does not recover failed messages, because a failed message short-circuits past it. To recover from an expected error under strict mode, wrap the fallible step and its recovery logic in a [`try_catch` processor](https://docs.redpanda.com/connect/components/processors/try_catch/). To retry transient failures until they succeed, use a [`retry` processor](https://docs.redpanda.com/connect/components/processors/retry/). Strict error handling requires Redpanda Connect 4.98.0 or later. > 📝 **NOTE** > > Strict error handling will become the default and only behavior in the next major version of Redpanda Connect. The rest of this topic describes error-handling patterns for the default behavior. ## [](#abandon-on-failure)Abandon on failure You can use the [`try` processor](https://docs.redpanda.com/connect/components/processors/try/) to define a list of processors that are executed in sequence. If a processor fails for a particular message, that message skips the remaining processors. For example: - If `processor_1` fails to process a message, that message skips `processor_2` and `processor_3`. - If a message is processed by `processor_1`, but `processor_2` fails, that message skips `processor_3`, and so on. ```yaml pipeline: processors: - try: - resource: processor_1 - resource: processor_2 # Skip if processor_1 fails - resource: processor_3 # Skip if processor_1 or processor_2 fails ``` ## [](#recover-failed-messages)Recover failed messages You can also route failed messages through defined processing steps using a [`catch` processor](https://docs.redpanda.com/connect/components/processors/catch/). For example, if `processor_1` fails to process a message, it is rerouted to `processor_2`. ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - catch: - resource: processor_2 # Processes rerouted messages ``` After messages complete all processing steps defined in the `catch` block, failure flags are removed and they are treated like regular messages. To keep failure flags in messages, you can simulate a `catch` block using a [`switch` processor](https://docs.redpanda.com/connect/components/processors/switch/): ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - switch: - check: errored() processors: - resource: processor_2 # Processes rerouted messages ``` ## [](#logging-errors)Logging errors When an error occurs, there may be useful information stored in the error flag. You can use [`error`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error) Bloblang function interpolations to write this information to logs. You can also add the following Bloblang functions to expose additional details about the processor that triggered the error. - [`error_source_label`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error_source_label) - [`error_source_name`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error_source_name) - [`error_source_path`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error_source_path) For example, this configuration catches processor failures and writes the following information to logs: - The label of the processor (`${!error_source_label()}`) that failed - The cause of the failure (`${!error()}`) ```yaml pipeline: processors: - try: - resource: processor_1 # Processor that might fail - resource: processor_2 # Processor that might fail - resource: processor_3 # Processor that might fail - catch: - log: message: "Processor ${!error_source_label()} failed due to: ${!error()}" ``` You could also add an error message to the message payload: ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - resource: processor_2 # Processor that might fail - resource: processor_3 # Processor that might fail - catch: - mapping: | root = this root.meta.error = error() ``` ### [](#logging-fatal-errors)Logging fatal errors During development and testing, you can use the [`crash` processor](https://docs.redpanda.com/connect/components/processors/crash/) to halt pipeline execution and log a fatal message when an unhandled error occurs. Use Bloblang functions to customize the message. For example: ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - catch: # Writes a custom fatal log message - crash: "Processing failed at ${!error_source_label()} due to: ${!error()}" ``` ## [](#attempt-until-success)Attempt until success To process a particular message until it is successful, try using a [`retry`](https://docs.redpanda.com/connect/components/processors/retry/) processor: ```yaml pipeline: processors: - retry: backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s processors: # Retries this processor until the message is processed, or the maximum elapsed time is reached. - resource: processor_1 ``` ## [](#drop-failed-messages)Drop failed messages To filter out any failed messages from your pipeline, you can use a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/): ```yaml pipeline: processors: - mapping: root = if errored() { deleted() } ``` The mapping uses the error flag to identify any failed messages in a batch and drops the messages, which propagates acknowledgements (also known as "acks") upstream to the pipeline’s input. ## [](#reject-messages)Reject messages Some inputs, such as `nats`, `gcp_pubsub`, and `amqp_1`, support nacking (rejecting) messages. Rather than delivering unprocessed messages to your output, you can use the [`reject_errored` output](https://docs.redpanda.com/connect/components/outputs/reject_errored/) to perform a nack (or rejection) on them: ```yaml output: reject_errored: resource: processor_1 # Only non-errored messages go here ``` ## [](#route-to-a-dead-letter-queue)Route to a dead-letter queue You can also route failed messages to a different output by nesting the [`reject_errored` output](https://docs.redpanda.com/connect/components/outputs/reject_errored/) within a [`fallback` output](https://docs.redpanda.com/connect/components/outputs/fallback/) ```yaml output: fallback: - reject_errored: resource: processor_1 # Only non-errored messages go here - resource: processor_2 # Only errored messages, or delivery failures to processor_1, go here ``` If you want to route data differently based on the type of error message, you can use a [`switch` output](https://docs.redpanda.com/connect/components/outputs/switch/): ```yaml output: switch: cases: # Capture specifically cat-related errors - check: errored() && error().contains("meow") output: resource: processor_1 # Capture all other errors - check: errored() output: resource: processor_2 # Finally, route all successfully processed messages here - output: resource: processor_3 ``` Finally, you can attach additional metadata when routing messages to the dead-letter queue, such as the error message. This can be done by running a series of [processors](https://docs.redpanda.com/connect/components/processors/about/) before sending the data to the final [output](https://docs.redpanda.com/connect/components/outputs/about/). ```yaml output: fallback: - reject_errored: resource: processor_1 # Only non-errored messages go here - processors: - mutation: | root.error = @fallback_error # Adds the error message before sending the message to the dead-letter queue output resource: processor_2 # Only errored messages, or delivery failures to processor_1, go here ``` --- # Page 335: Field Paths **URL**: https://docs.redpanda.com/connect/configuration/field_paths.md --- # Field Paths --- title: Field Paths latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: field_paths page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: field_paths.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/field_paths.adoc description: Learn the dot path syntax that Redpanda Connect components use to target fields within messages and configuration. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Many components within Redpanda Connect allow you to target certain fields using a JSON dot path. The syntax of a path within Redpanda Connect is similar to [JSON Pointers](https://tools.ietf.org/html/rfc6901), except with dot separators instead of slashes (and no leading dot.) When a path is used to set a value any path segment that does not yet exist in the structure is created as an object. For example, if we had the following JSON structure: ```json { "foo": { "bar": 21 } } ``` The query path `foo.bar` would return `21`. The characters `~` (%x7E) and `.` (%x2E) have special meaning in Redpanda Connect paths. Therefore `~` needs to be encoded as `~0` and `.` needs to be encoded as `~1` when these characters appear within a key. For example, if we had the following JSON structure: ```json { "foo.foo": { "bar~bo": { "": { "baz": 22 } } } } ``` The query path `foo~1foo.bar~0bo..baz` would return `22`. ## [](#arrays)Arrays When Redpanda Connect encounters an array while traversing a JSON structure it requires the next path segment to be either an integer of an existing index or, depending on whether the path is used to query or set the target value, the character `*` or `-` respectively. For example, if we had the following JSON structure: ```json { "foo": [ 0, 1, { "bar": 23 } ] } ``` The query path `foo.2.bar` would return `23`. ### [](#querying)Querying When a query reaches an array the character `*` indicates that the query should return the value of the remaining path from each array element (within an array.) ### [](#setting)Setting When an array is reached the character `-` indicates that a new element should be appended to the end of the existing elements, if this character is not the final segment of the path then an object is created. --- # Page 336: Interpolation **URL**: https://docs.redpanda.com/connect/configuration/interpolation.md --- # Interpolation --- title: Interpolation latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: interpolation page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: interpolation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/interpolation.adoc description: Learn how to use environment variables and function interpolations to insert dynamic values into Redpanda Connect configurations. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Redpanda Connect allows you to dynamically set config fields with environment variables anywhere within a config file using the syntax `${}` (or `${:}` in order to specify a default value). This is useful for setting environment specific fields such as addresses: ```yaml input: kafka: addresses: [ "${BROKERS}" ] consumer_group: redpanda_connect_consumer topics: [ "haha_business" ] ``` ```sh BROKERS="foo:9092,bar:9092" rpk connect run ./config.yaml ``` If a literal string is required that matches this pattern (`${foo}`) you can escape it with double brackets. For example, the string `${{foo}}` is read as the literal `${foo}`. ## [](#undefined-variables)Undefined variables When an environment variable interpolation is found within a config, does not have a default value specified, and the environment variable is not defined a linting error will be reported. In order to avoid this it is possible to specify environment variable interpolations with an explicit empty default value by adding the colon without a following value, i.e. `${FOO:}` would be equivalent to `${FOO}` and would not trigger a linting error should `FOO` not be defined. ## [](#yaml-tags)YAML tags By default, Redpanda Connect interpolates environment variables as strings. You can use [YAML tags](https://yaml.org/spec/1.2.2/#24-tags) to interpret values as another scalar type, such as integers. ```yaml output: redpanda: # ... batching: count: !!int ${BATCHING_COUNT:500} period: "${BATCHING_PERIOD:1s}" ``` Redpanda Connect supports the [core schema tags](https://yaml.org/spec/1.2.2/#103-core-schema) for scalar types: - `null` - `bool` - `int` - `float` - `str` (default) ## [](#bloblang-queries)Bloblang queries Some Redpanda Connect fields also support [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) function interpolations, which are much more powerful expressions that allow you to query the contents of messages and perform arithmetic. The syntax of a function interpolation is `${!}`, where the contents are a bloblang query (the right-hand-side of a bloblang map) including a range of [functions](https://docs.redpanda.com/connect/guides/bloblang/about/#functions). For example, with the following config: ```yaml output: kafka: addresses: [ "TODO:6379" ] topic: 'dope-${! json("topic") }' ``` A message with the contents `{"topic":"foo","message":"hello world"}` would be routed to the Kafka topic `dope-foo`. If a literal string is required that matches this pattern (`${!foo}`) then, similar to environment variables, you can escape it with double brackets. For example, the string `${{!foo}}` would be read as the literal `${!foo}`. Bloblang supports arithmetic, boolean operators, coalesce and mapping expressions. For more in-depth details about the language [check out the docs](https://docs.redpanda.com/connect/guides/bloblang/about/). ## [](#examples)Examples ### [](#reference-metadata)Reference metadata A common usecase for interpolated functions is dynamic routing at the output level using metadata: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("output_topic") } key: ${! meta("key") } ``` ### [](#coalesce-and-mapping)Coalesce and mapping Bloblang supports coalesce and mapping, which makes it easy to extract values from slightly varying data structures: ```yaml pipeline: processors: - cache: resource: foocache operator: set key: '${! json().message.(foo | bar).id }' value: '${! content() }' ``` Here’s a map of inputs to resulting values: {"foo":{"a":{"baz":"from\_a"},"c":{"baz":"from\_c"}}} -> from\_a {"foo":{"b":{"baz":"from\_b"},"c":{"baz":"from\_c"}}} -> from\_b {"foo":{"b":null,"c":{"baz":"from\_c"}}} -> from\_c ### [](#delayed-processing)Delayed processing We have a stream of JSON documents each with a unix timestamp field `doc.received_at` which is set when our platform receives it. We wish to only process messages an hour _after_ they were received. We can achieve this by running the `sleep` processor using an interpolation function to calculate the seconds needed to wait for: ```yaml pipeline: processors: - sleep: duration: '${! 3600 - ( timestamp_unix() - json("doc.created_at").number() ) }s' ``` If the calculated result is less than or equal to zero the processor does not sleep at all. If the value of `doc.created_at` is a string then our method `.number()` will attempt to parse it into a number. --- # Page 337: Metadata **URL**: https://docs.redpanda.com/connect/configuration/metadata.md --- # Metadata --- title: Metadata latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: metadata page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metadata.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/metadata.adoc description: Learn how message metadata works in Redpanda Connect, including how to get, set, and delete key/value pairs that travel with each message. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- In Redpanda Connect each message has raw contents and metadata, which is a map of key/value pairs representing an arbitrary amount of complementary data. When an input protocol supports attributes or metadata they will automatically be added to your messages, refer to the respective input documentation for a list of metadata keys. When an output supports attributes or metadata any metadata key/value pairs in a message will be sent (subject to service limits). ## [](#editing-metadata)Editing metadata Redpanda Connect allows you to add and remove metadata using the [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/). For example, you can do something like this in your pipeline: ```yaml pipeline: processors: - mapping: | # Remove all existing metadata from messages meta = deleted() # Add a new metadata field `time` from the contents of a JSON # field `event.timestamp` meta time = event.timestamp ``` You can also use [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) to delete individual metadata keys with: ```bloblang meta foo = deleted() ``` Or do more interesting things like remove all metadata keys with a certain prefix: ```bloblang meta = @.filter(kv -> !kv.key.has_prefix("kafka_")) ``` ## [](#using-metadata)Using metadata Metadata values can be referenced in any field that supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/). For example, you can route messages to Kafka topics using interpolation of metadata keys: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("target_topic") } ``` Redpanda Connect also allows you to conditionally process messages based on their metadata with the [`switch` processor](https://docs.redpanda.com/connect/components/processors/switch/): ```yaml pipeline: processors: - switch: - check: '@doc_type == "nested"' processors: - sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ foo, bar, baz ] args_mapping: | root = [ this.document.foo, this.document.bar, @kafka_topic, ] # In: {"document":{"foo":"value1","bar":"value2"}} ``` ## [](#restricting-metadata)Restricting metadata Outputs that support metadata, headers or some other variant of enriched fields on messages will attempt to send all metadata key/value pairs by default. However, sometimes it’s useful to refer to metadata fields at the output level even though we do not wish to send them with our data. In this case it’s possible to restrict the metadata keys that are sent with the field `metadata.exclude_prefixes` within the respective output config. For example, if we were sending messages to kafka using a metadata key `target_topic` to determine the topic but we wished to prevent that metadata key from being sent as a header we could use the following configuration: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("target_topic") } metadata: exclude_prefixes: - target_topic ``` And when the list of metadata keys that we do _not_ want to send is large it can be helpful to use a [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) in order to give all of these "private" keys a common prefix: ```yaml pipeline: processors: # Has an explicit list of public metadata keys, and everything else is given # an underscore prefix. - mapping: | let allowed_meta = [ "foo", "bar", "baz", ] meta = @.map_each_key(key -> if !$allowed_meta.contains(key) { "_" + key }) output: kafka: addresses: [ TODO ] topic: ${! meta("_target_topic") } metadata: exclude_prefixes: [ "_" ] ``` --- # Page 338: Processing Pipelines **URL**: https://docs.redpanda.com/connect/configuration/processing_pipelines.md --- # Processing Pipelines --- title: Processing Pipelines latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: processing_pipelines page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processing_pipelines.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/processing_pipelines.adoc description: Learn how the pipeline section applies processors to all messages and how to scale processing across parallel threads. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Within a Redpanda Connect configuration, in between `input` and `output`, is a `pipeline` section. This section describes an array of [processors](https://docs.redpanda.com/connect/components/processors/about/) that are to be applied to _all_ messages, and are not bound to any particular input or output. If you have processors that are heavy on CPU and aren’t specific to a certain input or output they are best suited for the pipeline section. It is advantageous to use the pipeline section as it allows you to set an explicit number of parallel threads of execution: ```yaml input: resource: foo pipeline: threads: 4 processors: - mapping: | root = this fans = fans.map_each(match { this.obsession > 0.5 => this _ => deleted() }) output: resource: bar ``` If the field `threads` is set to `-1` (the default) it will automatically match the number of logical CPUs available. By default almost all Redpanda Connect sources will utilize as many processing threads as have been configured, which makes horizontal scaling easy. --- # Page 339: Resources **URL**: https://docs.redpanda.com/connect/configuration/resources.md --- # Resources --- title: Resources latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: resources page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: resources.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/resources.adoc description: Learn how to declare components as named resources in Redpanda Connect so one configured instance can be referenced throughout a configuration. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- Resources are components within Redpanda Connect that are declared with a unique label and can be referenced any number of times within a configuration. Only one instance of each named resource is created, but it is safe to use it in multiple places as they can be shared without consequence. Some components such as caches and rate limits can _only_ be created as a resource. However, for components where it’s optional there are a few reasons why it might be advantageous to do so. ```yaml input: resource: foo pipeline: processors: - resource: bar - cache: operator: set resource: baz key: ${! json("id") } value: ${! content() } output: resource: buz input_resources: - label: foo file: paths: [ ./in.txt ] processor_resources: - label: bar mapping: 'root = content.lowercase()' cache_resources: - label: baz memory: default_ttl: 300s output_resources: - label: buz file: path: ./out.txt ``` ## [](#reusability)Reusability Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component as a resource. In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to "cleanse" the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource `get_foo`: ```yaml pipeline: processors: - resource: get_foo - catch: - mapping: | root = this root.content = this.content.strip_html() - resource: get_foo processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` ## [](#feature-toggling)Feature toggling ### [](#with-environment-variables)With environment variables There are two ways of using resources for feature toggling, the first is to define your feature components with unique names and then apply the old switcheroo with environment variables to select the one you wish to execute: ```yaml pipeline: processors: - resource: ${FEATURE_REQUEST} processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" - label: get_bar http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` Then when you execute Redpanda Connect use the environment variable to choose your resource: `FEATURE_REQUEST=get_foo rpk connect run ./your_config.yaml`. ### [](#with-imports)With imports However, Redpanda Connect allows you to import resources from separate files with the cli flag `-r` or `-resources`, which can be a useful way to switch out resources with common names based on your chosen environment. For example, with a main configuration file `config.yaml`: ```yaml pipeline: processors: - resource: get_foo ``` And then two resource files, one stored at the path `./staging/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` And another stored at the path `./production/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` We can select our chosen resource by changing which file we import, either running: ```bash rpk connect run -r ./staging/request.yaml ./config.yaml ``` Or: ```bash rpk connect run -r ./production/request.yaml ./config.yaml ``` These flags also support wildcards, which allows you to import an entire directory of resource files like `rpk connect run -r "./staging/*.yaml" ./config.yaml`. --- # Page 340: Secrets **URL**: https://docs.redpanda.com/connect/configuration/secrets.md --- # Secrets --- title: Secrets latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: secrets page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: secrets.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/secrets.adoc description: Learn how to add secrets to a Redpanda Connect configuration without exposing them. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- This topic outlines how to add secrets to a Redpanda Connect configuration without exposing them. ## [](#store-secrets-in-environment-variables)Store secrets in environment variables A common way to securely pass secrets to a service is to use environment variables. Redpanda Connect allows you to inject the values of environment variables into a configuration with the interpolation syntax `${SECRET}`. For example: ```yml thing: super_secret: "${SECRET}" ``` > 📝 **NOTE: Use quotes** > > Use quotes > > It is valid to have `super_secret: ${SECRET}` above (without the quotes), but if `SECRET` is unset then the configuration becomes structurally different. Therefore, it’s always best to wrap environment variable interpolations within quotes so that when the variable is unset you still have a valid configuration (with an empty string). For more information about this syntax, see the [interpolation field page](https://docs.redpanda.com/connect/configuration/interpolation/). ## [](#look-up-secrets-on-a-remote-system-at-runtime)Look up secrets on a remote system at runtime > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. Starting with version 4.39.0, you can use the `rpk connect` CLI flag `--secrets` to look up secrets values on a remote system at runtime (for example, in your secrets management solution). This means that Redpanda Connect resolves interpolations in your configuration at runtime without setting environment variables. For example, you could run the following command to retrieve the value for `"${SECRET}"`, when the secret is stored on a Redis server. ```bash rpk connect run ./config.yaml --secrets redis://secret:redis@example.com ``` The command tries to load the secret value from the specified Uniform Resource Name (URN) using the format `scheme://secret:server_address`. You can specify multiple URNs separated by commas, which are tried in turn until a secrets value is successfully returned. ### [](#supported-remote-systems)Supported remote systems You can retrieve secrets from all of the following remote systems. | Remote system | URN format | | --- | --- | | AWS Secret Manager | aws://region/prefixFor example: aws://eu-west-1/redpanda/ | | Azure Key Vault | az://vault-uri/prefix. The vault-uri value should not contain a https:// prefix. | | GCP Secret Manager | gcp://projectID/prefixFor example: gcp://project-id/redpanda- | | Redis | redis://secret:redis@example.com | ## [](#set-secrets-using-shell-commands)Set secrets using shell commands You can set specific fields within a configuration using the CLI flag `--set`, where the syntax is a `=` pair: - ``: A placeholder for the [dot-separated path to the field being set](https://docs.redpanda.com/connect/configuration/field_paths/). - ``: The value you want to set the field to. For example, to write data to an AWS DynamoDB table using the [`aws_dynamodb` output connector](https://docs.redpanda.com/connect/components/outputs/aws_dynamodb/), your configuration might look like this: ```yml output: label: "" aws_dynamodb: table: connect-db-test string_columns: id: ${!json("id")} full_content: ${!content()} batching: count: 5 period: 2m credentials: id: ACCESS_KEY_ID secret: SECRET_ACCESS_KEY ``` You could set your credentials to values stored within something like Hashicorp Vault by running the configuration with the `--set` flag and backticks to execute shell commands for the values: ```bash rpk connect run ./config.yaml \ --set "output.aws_dynamodb.credentials.id=`vault kv get -mount=secret access_key_id`" \ --set "output.aws_dynamodb.credentials.secret=`vault kv get -mount=secret secret_access_key`" ``` Using this method lets you inject the secret into the configuration without leaking it into an environment variable. ## [](#avoid-leaked-secrets)Avoid leaked secrets There are a few ways in which configurations parsed by Redpanda Connect can be exported back out of the service. In all of these cases Redpanda Connect will attempt to scrub any field values within the config that are known secrets (any field marked as a secret in the docs). However, if you’re embedding secrets within a configuration outside of the value of secret fields, maybe as part of a Bloblang mapping, then care should be made to avoid exposing the resulting configuration. This specifically means you should not enable [debug HTTP endpoints](https://docs.redpanda.com/connect/components/http/about/#debug-endpoints) when the port is exposed, and don’t use the `rpk connect echo` subcommand on configurations containing secrets unless you’re printing to a secure pipe. --- # Page 341: Telemetry **URL**: https://docs.redpanda.com/connect/configuration/telemetry.md --- # Telemetry --- title: Telemetry latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: telemetry page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: telemetry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/telemetry.adoc description: Learn what anonymized usage data the Redpanda Connect telemetry service sends, and how to inspect or disable it. page-git-created-date: "2024-11-05" page-git-modified-date: "2026-08-12" --- From version 4.38.0, the default Redpanda Connect configuration includes a telemetry service that sends anonymized usage statistics to Redpanda. Redpanda analyzes the data to find out how often each Redpanda Connect component is used in production environments, and common usage patterns. This data analysis aims to help: - Identify gaps in functionality. - Prioritize the delivery of new features, enhancements, and bug fixes. For example, if usage data shows that most `aws_s3` outputs are paired with a `mutation` processor, then embedding a `mutation` field in the component might be a useful enhancement. ## [](#data-collection-method)Data collection method When you execute a configuration, the Redpanda Connect instance sends a JSON payload to the collection server. The payload contains a high-level, anonymized summary of the contents of the configuration file. Field values are never transmitted, nor are decorations of the configuration, such as label names. For example, if you execute this configuration: ```yml input: label: message_input generate: interval: 1s mapping: 'root.input_field = "string"' output: label: message_output aws_s3: bucket: bucket_name path: filename.txt ``` Redpanda extracts the following details from the data collected: - A unique identifier for the Redpanda Connect instance. - How long the configuration has been running for. - Components used in the configuration. In this case, a `generate` input and an `aws_s3` output. - The IP address of the running Redpanda Connect instance. The code responsible for extracting usage data is available in the `connect` repository in the file: [./payload.go](https://github.com/redpanda-data/connect/blob/v4.38.0/internal/telemetry/payload.go). ## [](#data-collection-frequency)Data collection frequency To avoid analyzing instances used for experimentation or testing, a Redpanda Connect instance must have been running for at least 5 minutes before any usage data is collected. Once usage data starts to be emitted, it is sent to the collection server once every 24 hours. ## [](#disable-the-telemetry-service)Disable the telemetry service The telemetry service configuration is only included in the build artifacts released through [GitHub](https://github.com/redpanda-data/connect/releases) or Redpanda’s official [Docker images](https://hub.docker.com/r/redpandadata/connect/). This includes the `rpk connect` plugin, which is used to manage installations and upgrades of Redpanda Connect. Custom builds do not send usage data. To continue to use the official Redpanda artifacts or images but disable the telemetry service, you can either block internet traffic or run the following command: ```bash rpk connect run --disable-telemetry ``` Disabling telemetry does not affect the normal operation of Redpanda Connect. --- # Page 342: Templating **URL**: https://docs.redpanda.com/connect/configuration/templating.md --- # Templating --- title: Templating latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: templating page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: templating.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/templating.adoc description: Learn how templates work. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- > ⚠️ **CAUTION** > > Templates are an experimental feature and are subject to change outside major version releases. Templates are a way to define new Redpanda Connect components (similar to plugins) that are implemented by generating a Redpanda Connect configuration snippet from pre-defined parameter fields. This is useful when a common pattern of Redpanda Connect configuration is used but with varying parameters each time. A template is defined in a YAML file that can be imported when Redpanda Connect runs using the flag `-t`: ```bash rpk connect run -t "./templates/*.yaml" ./config.yaml ``` The template describes the type of the component and configuration fields that can be used to customize it, followed by a [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that translates an object containing those fields into a Redpanda Connect configuration structure. This allows you to use logic to generate more complex configurations: #### Template ```yaml name: aws_sqs_list type: input fields: - name: urls type: string kind: list - name: region type: string default: us-east-1 mapping: | root.broker.inputs = this.urls.map_each(url -> { "aws_sqs": { "url": url, "region": this.region, } }) ``` #### Configuration ```yaml input: aws_sqs_list: urls: - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1 - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2 pipeline: processors: - mapping: | root.id = uuid_v4() root.foo = this.inner.foo root.body = this.outer ``` #### Result ```yaml input: broker: inputs: - aws_sqs: url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1 region: us-east-1 - aws_sqs: url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2 region: us-east-1 pipeline: processors: - mapping: | root.id = uuid_v4() root.foo = this.inner.foo root.body = this.outer ``` You can see more examples of templates on [GitHub](https://github.com/redpanda-data/connect/blob/main/config/template_examples). ## [](#enum-field-options)Enum field options When `fields[].type` is set to `string_enum` or `string_annotated_enum`, the `fields[].options` field is required: - For `string_enum`, provide a non-empty array of string values. - For `string_annotated_enum`, provide a map of `value: "description"` pairs. The descriptions annotate the behavior of each value in the documentation. ```yaml fields: - name: mode type: string_enum options: ["fast", "balanced", "safe"] - name: delivery type: string_annotated_enum options: at_least_once: "May duplicate messages but never loses them." exactly_once: "Stronger guarantees but with a higher cost and constraints." ``` ## [](#test-a-template)Test a template Use the `tests` field to define unit tests for a template. The following examples show a test template, an associated test configuration, and the commands that run the tests. `test.template.yaml` ```yaml name: basictemplate type: input mapping: | root.generate = { "count": 1, "mapping": "root = \"" + @label.or("") + "\"" } tests: - name: basictemplate test label: quack config: {} expected: generate: count: 1 mapping: root = "quack" ``` `testconfig.yaml` ```yaml input: label: meow basictemplate: {} ``` To lint the template and run its tests, then run the test configuration with the template, use `rpk connect template lint` and `rpk connect run`: ```bash rpk connect template lint test.template.yaml rpk connect run -t "./test.template.yaml" ./testconfig.yaml ``` ## [](#fields)Fields ### [](#categories)`categories[]` An optional list of tags, which are used for arbitrarily grouping components in documentation. **Type**: `array` **Default**: `[]` ### [](#description)`description` A longer form description of the component and how to use it. **Type**: `string` **Default**: `""` ### [](#fields-2)`fields[]` The configuration fields of the template, fields specified here will be parsed from a Redpanda Connect config and will be accessible from the template mapping. **Type**: `array` ### [](#fields-advanced)`fields[].advanced` Whether this field is considered advanced. **Type**: `bool` **Default**: `false` ### [](#fields-default)`fields[].default` An optional default value for the field. If a default value is not specified then a configuration without the field is considered incorrect. **Type**: `unknown` ### [](#fields-description)`fields[].description` A description of the field. **Type**: `string` **Default**: `""` ### [](#fields-kind)`fields[].kind` The kind of the field. **Type**: `string` **Default**: `scalar` **Options**: `scalar`, `map`, `list` ### [](#fields-name)`fields[].name` The name of the field. **Type**: `string` ### [](#fields-options)`fields[].options` List of options for `string_enum` fields or map of annotated options for `string_annotated_enum` fields **Type**: `unknown` ### [](#fields-type)`fields[].type` The scalar type of the field. **Type**: `string` | Option | Summary | | --- | --- | | string | standard string type | | string_enum | string type which can have one of a discrete list of values | | string_annotated_enum | string type which can have one of a discrete list of values, where each value must be accompanied by a description that annotates its behaviour in the documentation | | int | standard integer type | | float | standard float type | | bool | standard boolean type | | bloblang | bloblang mapping | | unknown | allows for nesting arbitrary configuration inside of a field | ### [](#mapping)`mapping` A [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) mapping that translates the fields of the template into a valid Redpanda Connect configuration for the target component type. **Type**: `string` ### [](#metrics_mapping)`metrics_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/) that allows you to rename or prevent certain metrics paths from being exported. For more information check out the [metrics documentation](https://docs.redpanda.com/connect/components/metrics/about/#metric-mapping). When metric paths are created, renamed and dropped a trace log is written, enabling TRACE level logging is therefore a good way to diagnose path mappings. Invocations of this mapping are able to reference a variable $label in order to obtain the value of the label provided to the template config. This allows you to match labels with the root of the config. **Type**: `string` **Default**: `""` ```yaml # Examples: metrics_mapping: this.replace("input", "source").replace("output", "sink") # --- metrics_mapping: |- root = if ![ "input_received", "input_latency", "output_sent" ].contains(this) { deleted() } ``` ### [](#name)`name` The name of the component this template will create. **Type**: `string` ### [](#status)`status` The stability of the template describing the likelihood that the configuration spec of the template, or it’s behavior, will change. **Type**: `string` **Default**: `stable` | Option | Summary | | --- | --- | | stable | This template is stable and will therefore not change in a breaking way outside of major version releases. | | beta | This template is beta and will therefore not change in a breaking way unless a major problem is found. | | experimental | This template is experimental and therefore subject to breaking changes outside of major version releases. | | deprecated | This template has been deprecated and should no longer be used. | ### [](#summary)`summary` A short summary of the component. **Type**: `string` **Default**: `""` ### [](#tests)`tests[]` Optional unit test definitions for the template that verify certain configurations produce valid configs. These tests are executed with the command `rpk connect template lint`. **Type**: `array` **Default**: `[]` ### [](#tests-config)`tests[].config` A configuration to run this test with, the config resulting from applying the template with this config will be linted. **Type**: `object` ### [](#tests-expected)`tests[].expected` An optional configuration describing the expected result of applying the template, when specified the result will be diffed and any mismatching fields will be reported as a test error. **Type**: `object` ### [](#tests-label)`tests[].label` A label to assign to this template when running the test. **Type**: `string` **Default**: `""` ### [](#tests-name)`tests[].name` A name to identify the test. **Type**: `string` ### [](#type)`type` The type of the component this template will create. **Type**: `string` **Options**: `cache`, `input`, `output`, `processor`, `rate_limit` --- # Page 343: Unit Testing **URL**: https://docs.redpanda.com/connect/configuration/unit_testing.md --- # Unit Testing --- title: Unit Testing latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: unit_testing page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: unit_testing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/unit_testing.adoc description: Learn how to write and run unit tests for Redpanda Connect configuration files with the rpk connect test command. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- The Redpanda Connect service offers a command `rpk connect test` for running unit tests on sections of a configuration file. This makes it easy to protect your config files from regressions over time. ## [](#writing-a-test)Writing a test Let’s imagine we have a configuration file `foo.yaml` containing some processors: ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: '"%vend".format(content().uppercase().string())' output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` One way to write our unit tests for this config is to accompany it with a file of the same name and extension but suffixed with `_benthos_test`, which in this case would be `foo_benthos_test.yaml`. ```yml tests: - name: example test target_processors: '/pipeline/processors' environment: {} input_batch: - content: 'example content' metadata: example_key: example metadata value output_batches: - - content_equals: EXAMPLE CONTENTend metadata_equals: example_key: example metadata value ``` Under `tests` we have a list of any number of unit tests to execute for the config file. Each test is run in complete isolation, including any resources defined by the config file. Tests should be allocated a unique `name` that identifies the feature being tested. The field `target_processors` is either the label of a processor to test, or a [JSON Pointer](https://tools.ietf.org/html/rfc6901) that identifies the position of a processor, or list of processors, within the file which should be executed by the test. For example a value of `foo` would target a processor with the label `foo`, and a value of `/input/processors` would target all processors within the input section of the config. The field `environment` allows you to define an object of key/value pairs that set environment variables to be evaluated during the parsing of the target config file. These are unique to each test, allowing you to test different environment variable interpolation combinations. The field `input_batch` lists one or more messages to be fed into the targeted processors as a batch. Each message of the batch may have its raw content defined as well as metadata key/value pairs. For the common case where the messages are in JSON format, you can use `json_content` instead of `content` to specify the message structurally rather than verbatim. The field `output_batches` lists any number of batches of messages which are expected to result from the target processors. Each batch lists any number of messages, each one defining [`conditions`](#output-conditions) to describe the expected contents of the message. If the number of batches defined does not match the resulting number of batches the test will fail. If the number of messages defined in each batch does not match the number in the resulting batches the test will fail. If any condition of a message fails then the test fails. ### [](#inline-tests)Inline tests Sometimes it’s more convenient to define your tests within the config being tested. This is fine, simply add the `tests` field to the end of the config being tested. ### [](#bloblang-tests)Bloblang tests Sometimes when working with large [Bloblang mappings](https://docs.redpanda.com/connect/guides/bloblang/about/) it’s preferred to have the full mapping in a separate file to your Redpanda Connect configuration. In this case it’s possible to write unit tests that target and execute the mapping directly with the field `target_mapping`, which when specified is interpreted as either an absolute path or a path relative to the test definition file that points to a file containing only a Bloblang mapping. For example, if we were to have a file `cities.blobl` containing a mapping: ```bloblang root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` We can accompany it with a test file `cities_test.yaml` containing a regular test definition: ```yml tests: - name: test cities mapping target_mapping: './cities.blobl' environment: {} input_batch: - content: | { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } output_batches: - - json_equals: {"Cities": "Bellevue, Olympia, Seattle"} ``` And execute this test the same way we execute other Redpanda Connect tests (`rpk connect test ./dir/cities_test.yaml`, `rpk connect test ./dir/…​`, etc). ### [](#fragmented-tests)Fragmented tests Sometimes the number of tests you need to define in order to cover a config file is so vast that it’s necessary to split them across multiple test definition files. This is possible but Redpanda Connect still requires a way to detect the configuration file being targeted by these fragmented test definition files. In order to do this we must prefix our `target_processors` field with the path of the target relative to the definition file. The syntax of `target_processors` in this case is a full [JSON Pointer](https://tools.ietf.org/html/rfc6901) that should look something like `target.yaml#/pipeline/processors`. For example, if we saved our test definition above in an arbitrary location like `./tests/first.yaml` and wanted to target our original `foo.yaml` config file, we could do that with the following: ```yml tests: - name: example test target_processors: '../foo.yaml#/pipeline/processors' environment: {} input_batch: - content: 'example content' metadata: example_key: example metadata value output_batches: - - content_equals: EXAMPLE CONTENTend metadata_equals: example_key: example metadata value ``` ## [](#input-definitions)Input Definitions ### [](#content)`content` Sets the raw content of the message. ### [](#json_content)`json_content` ```yml json_content: foo: foo value bar: [ element1, 10 ] ``` Sets the raw content of the message to a JSON document matching the structure of the value. ### [](#file_content)`file_content` ```yml file_content: ./foo/bar.txt ``` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. ### [](#metadata)`metadata` A map of key/value pairs that sets the metadata values of the message. ## [](#output-conditions)Output Conditions ### [](#bloblang)`bloblang` ```yml bloblang: 'this.age > 10 && @foo.length() > 0' ``` Executes a [Bloblang expression](https://docs.redpanda.com/connect/guides/bloblang/about/) on a message, if the result is anything other than a boolean equalling `true` the test fails. ### [](#content_equals)`content_equals` ```yml content_equals: example content ``` Checks the full raw contents of a message against a value. ### [](#content_matches)`content_matches` ```yml content_matches: "^foo [a-z]+ bar$" ``` Checks whether the full raw contents of a message matches a regular expression (re2). ### [](#metadata_equals)`metadata_equals` ```yml metadata_equals: example_key: example metadata value ``` Checks a map of metadata keys to values against the metadata stored in the message. If there is a value mismatch between a key of the condition versus the message metadata this condition will fail. ### [](#file_equals)`file_equals` ```yml file_equals: ./foo/bar.txt ``` Checks that the contents of a message matches the contents of a file. The path of the file should be relative to the path of the test file. ### [](#file_json_equals)`file_json_equals` ```yml file_json_equals: ./foo/bar.json ``` Checks that both the message and the file contents are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. ### [](#json_equals)`json_equals` ```yml json_equals: { "key": "value" } ``` Checks that both the message and the condition are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. You can also structure the condition content as YAML and it will be converted to the equivalent JSON document for testing: ```yml json_equals: key: value ``` ### [](#json_contains)`json_contains` ```yml json_contains: { "key": "value" } ``` Checks that both the message and the condition are valid JSON documents, and that the message is a superset of the condition. ## [](#running-tests)Running tests Executing tests for a specific config can be done by pointing the subcommand `test` at either the config to be tested or its test definition, e.g. `rpk connect test ./config.yaml` and `rpk connect test ./config_benthos_test.yaml` are equivalent. The `test` subcommand also supports wildcard patterns e.g. `rpk connect test ./foo/*.yaml` will execute all tests within matching files. In order to walk a directory tree and execute all tests found you can use the shortcut `./…​`, e.g. `rpk connect test ./…​` will execute all tests found in the current directory, any child directories, and so on. If you want to allow components to write logs at a provided level to stdout when running the tests, you can use `rpk connect test --log `. Please consult the [logger docs](https://docs.redpanda.com/connect/components/logger/about/) for further details. ## [](#mocking-processors)Mocking processors BETA: This feature is currently in a BETA phase, which means breaking changes could be made if a fundamental issue with the feature is found. Sometimes you’ll want to write tests for a series of processors, where one or more of them are networked (or otherwise stateful). Rather than creating and managing mocked services you can define mock versions of those processors in the test definition. For example, if we have a config with the following processors: ```yaml pipeline: processors: - mapping: 'root = "simon says: " + content()' - label: get_foobar_api http: url: http://example.com/foobar verb: GET - mapping: 'root = content().uppercase()' ``` Rather than create a fake service for the `http` processor to interact with we can define a mock in our test definition that replaces it with a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/). Mocks are configured as a map of labels that identify a processor to replace and the config to replace it with: ```yaml tests: - name: mocks the http proc target_processors: '/pipeline/processors' mocks: get_foobar_api: mapping: 'root = content().string() + " this is some mock content"' input_batch: - content: "hello world" output_batches: - - content_equals: "SIMON SAYS: HELLO WORLD THIS IS SOME MOCK CONTENT" ``` With the above test definition the `http` processor will be swapped out for `mapping: 'root = content().string() + " this is some mock content"'`. For the purposes of mocking it is recommended that you use a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) that simply mutates the message in a way that you would expect the mocked processor to. > 📝 **NOTE** > > It’s not currently possible to mock components that are imported as separate resource files (using `--resource`/`-r`). It is recommended that you mock these by maintaining separate definitions for test purposes (`-r "./test/*.yaml"`). ### [](#more-granular-mocking)More granular mocking It is also possible to target specific fields within the test config by [JSON pointers](https://tools.ietf.org/html/rfc6901) as an alternative to labels. The following test definition would create the same mock as the previous: ```yaml tests: - name: mocks the http proc target_processors: '/pipeline/processors' mocks: /pipeline/processors/1: mapping: 'root = content().string() + " this is some mock content"' input_batch: - content: "hello world" output_batches: - - content_equals: "SIMON SAYS: HELLO WORLD THIS IS SOME MOCK CONTENT" ``` ## [](#fields)Fields The schema of a template file is as follows: ### [](#tests)`tests` A list of one or more unit tests to execute. **Type**: `array` ### [](#tests-name)`tests[].name` The name of the test, this should be unique and give a rough indication of what behavior is being tested. **Type**: `string` ### [](#tests-environment)`tests[].environment` An optional map of environment variables to set for the duration of the test. **Type**: `object` ### [](#tests-target_processors)`tests[].target_processors` A \[JSON Pointer\]\[json-pointer\] that identifies the specific processors which should be executed by the test. The target can either be a single processor or an array of processors. Alternatively a resource label can be used to identify a processor. It is also possible to target processors in a separate file by prefixing the target with a path relative to the test file followed by a # symbol. **Type**: `string` **Default**: `"/pipeline/processors"` ```yml # Examples target_processors: foo_processor target_processors: /pipeline/processors/0 target_processors: target.yaml#/pipeline/processors target_processors: target.yaml#/pipeline/processors ``` ### [](#tests-target_mapping)`tests[].target_mapping` A file path relative to the test definition path of a Bloblang file to execute as an alternative to testing processors with the `target_processors` field. This allows you to define unit tests for Bloblang mappings directly. **Type**: `string` **Default**: `""` ### [](#tests-mocks)`tests[].mocks` An optional map of processors to mock. Keys should contain either a label or a JSON pointer of a processor that should be mocked. Values should contain a processor definition, which will replace the mocked processor. Most of the time you’ll want to use a \[`mapping` processor\]\[processors.mapping\] here, and use it to create a result that emulates the target processor. **Type**: `object` ```yml # Examples mocks: get_foobar_api: mapping: root = content().string() + " this is some mock content" mocks: /pipeline/processors/1: mapping: root = content().string() + " this is some mock content" ``` ### [](#tests-input_batch)`tests[].input_batch` Define a batch of messages to feed into your test, specify either an `input_batch` or a series of `input_batches`. **Type**: `array` ### [](#tests-input_batch-content)`tests[].input_batch[].content` The raw content of the input message. **Type**: `string` ### [](#tests-input_batch-json_content)`tests[].input_batch[].json_content` Sets the raw content of the message to a JSON document matching the structure of the value. **Type**: `object` ```yml # Examples json_content: bar: - element1 - 10 foo: foo value ``` ### [](#tests-input_batch-file_content)`tests[].input_batch[].file_content` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_content: ./foo/bar.txt ``` ### [](#tests-input_batch-metadata)`tests[].input_batch[].metadata` A map of metadata key/values to add to the input message. **Type**: `object` ### [](#tests-input_batches)`tests[].input_batches` Define a series of batches of messages to feed into your test, specify either an `input_batch` or a series of `input_batches`. **Type**: `two-dimensional array` ### [](#tests-input_batches-content)`tests[].input_batches[][].content` The raw content of the input message. **Type**: `string` ### [](#tests-input_batches-json_content)`tests[].input_batches[][].json_content` Sets the raw content of the message to a JSON document matching the structure of the value. **Type**: `object` ```yml # Examples json_content: bar: - element1 - 10 foo: foo value ``` ### [](#tests-input_batches-file_content)`tests[].input_batches[][].file_content` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_content: ./foo/bar.txt ``` ### [](#tests-input_batches-metadata)`tests[].input_batches[][].metadata` A map of metadata key/values to add to the input message. **Type**: `object` ### [](#tests-output_batches)`tests[].output_batches` List of output batches. **Type**: `two-dimensional array` ### [](#tests-output_batches-bloblang)`tests[].output_batches[][].bloblang` Executes a Bloblang mapping on the output message, if the result is anything other than a boolean equalling `true` the test fails. **Type**: `string` ```yml # Examples bloblang: this.age > 10 && @foo.length() > 0 ``` ### [](#tests-output_batches-content_equals)`tests[].output_batches[][].content_equals` Checks the full raw contents of a message against a value. **Type**: `string` ### [](#tests-output_batches-content_matches)`tests[].output_batches[][].content_matches` Checks whether the full raw contents of a message matches a regular expression (re2). **Type**: `string` ```yml # Examples content_matches: ^foo [a-z]+ bar$ ``` ### [](#tests-output_batches-metadata_equals)`tests[].output_batches[][].metadata_equals` Checks a map of metadata keys to values against the metadata stored in the message. If there is a value mismatch between a key of the condition versus the message metadata this condition will fail. **Type**: `object` ```yml # Examples metadata_equals: example_key: example metadata value ``` ### [](#tests-output_batches-file_equals)`tests[].output_batches[][].file_equals` Checks that the contents of a message matches the contents of a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_equals: ./foo/bar.txt ``` ### [](#tests-output_batches-file_json_equals)`tests[].output_batches[][].file_json_equals` Checks that both the message and the file contents are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_json_equals: ./foo/bar.json ``` ### [](#tests-output_batches-json_equals)`tests[].output_batches[][].json_equals` Checks that both the message and the condition are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. **Type**: `object` ```yml # Examples json_equals: key: value ``` ### [](#tests-output_batches-json_contains)`tests[].output_batches[][].json_contains` Checks that both the message and the condition are valid JSON documents, and that the message is a superset of the condition. **Type**: `object` ```yml # Examples json_contains: key: value ``` ### [](#tests-output_batches-file_json_contains)`tests[].output_batches[][].file_json_contains` Checks that both the message and the file contents are valid JSON documents, and that the message is a superset of the condition. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_json_contains: ./foo/bar.json ``` --- # Page 344: Using CUE **URL**: https://docs.redpanda.com/connect/configuration/using_cue.md --- # Using CUE --- title: Using CUE latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: using_cue page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: using_cue.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/using_cue.adoc description: Learn how to use CUE to define, type-check, and generate Redpanda Connect configurations. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-12" --- > ⚠️ **CAUTION** > > CUE support is experimental. It may change for some time to improve CUE’s ability to type-check Redpanda Connect configurations at the expense of causing new validation errors when moving from one Redpanda Connect release to the next. [**CUE**](https://cuelang.org/) is a powerful configuration language that makes it easier and safer to build Redpanda Connect configurations. It achieves this by validating and type-checking configurations as well as allowing you to build useful utilities that reduce boilerplate. In this guide, we will see how to build a Redpanda Connect configuration using CUE, export it to YAML and execute it. ## [](#prerequisites)Prerequisites Before you get started, ensure that you have installed CUE by [following this guide](https://cuelang.org/docs/install/). If this is your first time workin with it, then it’s a great idea to step through the [CUE tutorial](https://cuelang.org/docs/tutorial/) and familiarize yourself with the language. ## [](#create)Create Create a directory for the CUE module that will contain the configuration: ```shell mkdir hello-cue cd hello-cue cue mod init example.com/hello-cue touch config.cue ``` > CUE modules must start with a hostname. This will typically be the URL of your repository. For example: `cue mod init github.com/redpanda-data/hello-cue`. The `rpk connect list` command will generate a CUE package containing the types we’ll need to build our configuration. Let’s write this package into our project: ```bash mkdir benthos rpk connect list --format cue > benthos/schema.cue ``` At this point, you should now have the following directory structure: hello-cue/ benthos/ schema.cue cue.mod/ pkg/ usr/ module.cue config.cue We are now ready to write our Redpanda Connect config in CUE. Let’s start by editing our `config.cue` to include the following snippet: ```cue import "example.com/hello-cue/benthos" benthos.#Config & { input: { generate: { mapping: """ root = { "message": "Hello, CUE!" } """ } } pipeline: { processors: [ { mapping: """ root = this root.id = uuid_v4() """ } ] } output: { stdout: {} } } ``` Let’s see what this will look like as YAML by running `cue export` while in the `hello-cue` directory: ```shell cue export --out yaml config.cue ``` This should output something like this: ```yaml input: generate: mapping: 'root = { "message": "Hello, CUE!" }' pipeline: processors: - mapping: |- root = this root.id = uuid_v4() output: stdout: {} tests: [] ``` We can run this with Redpanda Connect to see that it indeed works: ```bash rpk connect run <(cue export --out yaml config.cue) ``` When you are satisfied with the results, terminate the Redpanda Connect process and let’s move on to look at some of the nice features that we get with CUE. ## [](#enhance)Enhance The `config.cue` above looks eerily like JSON. This is because CUE is a superset of JSON and shares its syntax. However, we can shorten our configuration to reduce indentation and curly brackets. Let’s rewrite `config.cue` to look like this: ```cue import "example.com/hello-cue/benthos" benthos.#Config & { input: generate: mapping: """ root = { "message": "Hello, CUE!" } """ pipeline: processors: [ { mapping: """ root = this root.id = uuid_v4() """ } ] output: stdout: {} } ``` If you run the same `cue export` command from earlier, you’ll notice that the YAML output is the same. Next, we’ll look at what some error handling patterns might look like with CUE. One typical technique to detect messages with errors is to use the `switch` output to wrap another output with some error detection and handling. Another pattern involves limiting the number of retries on a given output that is misbehaving and rejecting or dropping messages with some useful logging. If we combine all these concepts together we get: ```yaml output: switch: cases: - check: errored() output: reject: "failed to process message: ${! error() }" - output: retry: max_retries: 5 output: gcp_pubsub: project: "sample-project" topic: "sample-topic" ``` There are quite a few lines of YAML here and we seem to be going sideways as we compose more functionality. We can try and make this more manageable with CUE! Let’s create a new file in our `hello-cue` directory called `benthos/helpers.cue`: ```shell touch benthos/helpers.cue ``` In this file, add the following snippet: ```cue package benthos #Guarded: self = { // The desired output that will be wrapped with error handling mechanisms #output: #Output // The error text to emit if the output receives any messages which contained // processing errors #errorMessage: string // The number of retries to attempt on the desired output (default is 3) #maxRetries: uint | *3 // The error message to emit if the retry attempts are exhausted #retryErrorMessage: string // Whether to drop or reject any failed messages #errorHandling: "drop" | "reject" switch: cases: [ { check: "errored()" output: { if self.#errorHandling == "reject" { reject: self.#errorMessage } if self.#errorHandling == "drop" { drop: {} processors: [{ log: message: self.#errorMessage }] } } }, { output: fallback: [ { retry: { max_retries: self.#maxRetries output: self.#output } }, { if self.#errorHandling == "reject" { reject: self.#retryErrorMessage } if self.#errorHandling == "drop" { drop: {} processors: [{ log: message: self.#retryErrorMessage }] } } ] } ] } ``` Now, let’s get back to `config.cue` and edit a few bits while leveraging this helper: ```cue import "example.com/hello-cue/benthos" benthos.#Config & { input: generate: { count: 1 interval: "0" mapping: """ root = { "message": "Hello, CUE!" } """ } output: benthos.#Guarded & { #errorMessage: "failed to process message: ${! error() }" #maxRetries: 3 #retryErrorMessage: "failed to output message after \(#maxRetries) retries" #errorHandling: "drop" #output: http_client: { url: "http://localhost:4195/sad-blob" retries: 0 } } } ``` If you rerun `cue export` now, you’ll see that we’ve wrapped our output with a couple of error handling mechanisms. We also had access to powerful CUE features like conditional fields based on `#errorHandling`, default values and interpolations. ```yaml input: generate: count: 1 interval: "0" mapping: 'root = { "message": "Hello, CUE!" }' output: switch: cases: - check: errored() output: drop: {} processors: - log: message: 'failed to process message: ${! error() }' - output: fallback: - retry: max_retries: 3 output: http_client: url: http://localhost:4195/sad-blob retries: 0 - drop: {} processors: - log: message: failed to output message after 3 retries tests: [] ``` The final directory structure of your hello-cue project should look like this: hello-cue/ benthos/ schema.cue helpers.cue cue.mod/ pkg/ usr/ module.cue config.cue ## [](#included-cue-types)Included CUE types The `benthos.cue` file we emitted earlier contains a number of useful types that we can use when build configuration files and helpers. These include: - `benthos.#Config` This definition describes the format of a Redpanda Connect config file. You’ll want to use it at the top of your configuration file to validate its overall structure. - `benthos.#Input` - `benthos.#Output` - `benthos.#Processor` - `benthos.#RateLimit` - `benthos.#Buffer` - `benthos.#Cache` - `benthos.#Metric` - `benthos.#Tracer` Each of these definitions is a disjunction that holds all the corresponding components in Redpanda Connect. In other words, a CUE field that is specified as `benthos.#Input`, such as `myfield: benthos.#Input`, must resolve to a valid Redpanda Connect input. ## [](#wrap-up)Wrap up Being able to define helper packages and definitions like `#Guarded` and reusing them across your Redpanda Connect configurations is a really powerful feature of CUE. This will allow you to share consistent good practices without messy boilerplate across projects and teams! --- # Page 345: Windowed Processing **URL**: https://docs.redpanda.com/connect/configuration/windowed_processing.md --- # Windowed Processing --- title: Windowed Processing latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: windowed_processing page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: windowed_processing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/windowed_processing.adoc description: Learn how to process periodic windows of messages with Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- A window is a batch of messages made with respect to time, with which we are able to perform processing that can analyze or aggregate the messages of the window. This is useful in stream processing as the dataset is never "complete", and therefore in order to perform analysis against a collection of messages we must do so by creating a continuous feed of windows (collections), where our analysis is made against each window. For example, given a stream of messages relating to cars passing through various traffic lights: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T09:49:35Z", "registration_plate": "AB1C DEF", "passengers": 3 } ``` Windowing allows us to produce a stream of messages representing the total traffic for each light every hour: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T10:00:00Z", "unique_cars": 15, "passengers": 43 } ``` ## [](#creating-windows)Creating windows The first step in processing windows is producing the windows themselves, this can be done by configuring a window producing buffer after your input: ### System A `system_window` buffer creates windows by following the system clock of the running machine. Windows will be created and emitted at predictable times, but this also means windows for historic data will not be emitted and therefore prevents backfills of traffic data: ```yaml input: kafka: addresses: [ TODO ] topics: [ traffic_data ] consumer_group: traffic_consumer checkpoint_limit: 1000 buffer: system_window: timestamp_mapping: root = this.created_at size: 1h allowed_lateness: 3m ``` For more information about this buffer refer to the `system_window` buffer docs. ## [](#grouping)Grouping With a window buffer chosen our stream of messages will be emitted periodically as batches of all messages that fit within each window. Since we want to analyse the window separately for each traffic light we need to expand this single batch out into one for each traffic light identifier within the window. For that purpose we have two processor options: [`group_by`](https://docs.redpanda.com/connect/components/processors/group_by/) and [`group_by_value`](https://docs.redpanda.com/connect/components/processors/group_by_value/). In our case we want to group by the value of the field `traffic_light` of each message, which we can do with the following: ```yaml pipeline: processors: - group_by_value: value: ${! json("traffic_light") } ``` ## [](#aggregating)Aggregating Once our window has been grouped the next step is to calculate the aggregated passenger and unique cars counts. For this purpose the Redpanda Connect [mapping language Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) comes in handy as the method [`from_all`](https://docs.redpanda.com/connect/guides/bloblang/methods/#from_all) executes the target function against the entire batch and returns an array of the values, allowing us to mutate the result with chained methods such as [`sum`](https://docs.redpanda.com/connect/guides/bloblang/methods/#sum): ```yaml pipeline: processors: - group_by_value: value: ${! json("traffic_light") } - mapping: | let is_first_message = batch_index() == 0 root.traffic_light = this.traffic_light root.created_at = @window_end_timestamp root.total_cars = if $is_first_message { json("registration_plate").from_all().unique().length() } root.passengers = if $is_first_message { json("passengers").from_all().sum() } # Only keep the first batch message containing the aggregated results. root = if ! $is_first_message { deleted() } ``` [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) is very powerful, and by using [`from`](https://docs.redpanda.com/connect/guides/bloblang/methods/#from) and [`from_all`](https://docs.redpanda.com/connect/guides/bloblang/methods/#from_all) it’s possible to perform a wide range of batch-wide processing. If you fancy a challenge try updating the above mapping to only count passengers from the first journey of each registration plate in the window (hint: the [`fold` method](https://docs.redpanda.com/connect/guides/bloblang/methods/#fold) might come in handy). --- # Page 346: Cookbooks **URL**: https://docs.redpanda.com/connect/cookbooks.md --- # Cookbooks --- title: Cookbooks latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/index.adoc description: Explore step-by-step cookbooks that walk through real-world Redpanda Connect configurations. page-git-created-date: "2024-06-03" page-git-modified-date: "2026-08-12" --- - [DynamoDB CDC Patterns](dynamodb_cdc/) Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect. - [Oracle Database CDC Patterns](oracledb_cdc/) Learn how to capture, filter, transform, and route Oracle Database change data capture (CDC) events with Redpanda Connect. - [MySQL CDC Patterns](mysql_cdc/) Learn how to capture, filter, transform, and route MySQL change data capture (CDC) events with Redpanda Connect. - [PostgreSQL CDC Patterns](postgres_cdc/) Learn how to capture, filter, transform, and route PostgreSQL change data capture (CDC) events with Redpanda Connect. - [MongoDB CDC Patterns](mongodb_cdc/) Learn how to capture, filter, transform, and route MongoDB change data capture (CDC) events with Redpanda Connect. - [SQL Server CDC Patterns](microsoft_sql_server_cdc/) Learn how to capture, filter, transform, and route Microsoft SQL Server change data capture (CDC) events with Redpanda Connect. - [TigerBeetle CDC Patterns](tigerbeetle_cdc/) Learn how to capture, filter, transform, and route TigerBeetle change data capture (CDC) events with Redpanda Connect. - [Google Cloud Spanner CDC Patterns](gcp_spanner_cdc/) Learn how to capture, filter, transform, and route Google Cloud Spanner change data capture (CDC) events with Redpanda Connect. - [CDC Schema Registry Patterns](cdc_schema_registry/) Register schemas from database change data capture (CDC) events in a schema registry and stream schema-encoded messages to Redpanda with Redpanda Connect. - [Enrichment Workflows](enrichments/) How to configure Redpanda Connect to process a workflow of enrichment services. - [Filtering and Sampling](filtering/) Configure Redpanda Connect to conditionally drop messages. - [Message Routing Patterns](message_routing/) Learn how to route, filter, and discard messages in Redpanda Connect pipelines. - [Work with Jira Issues](jira/) Learn how to use the Jira processor as both an input and output in Redpanda Connect. - [Joining Streams](joining_streams/) How to hydrate documents by joining multiple streams. - [Stream data to BigQuery with the Storage Write API](gcp_bigquery_write_api/) Configure Redpanda Connect to stream, upsert, and delete data in BigQuery using the Storage Write API. - [Ingest data into Snowflake](snowflake_ingestion/) Configure Redpanda Connect to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. - [Redpanda Migrator](redpanda_migrator/) Move your workloads from any Kafka system to Redpanda using a single command. Redpanda Migrator lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. It's Redpanda's alternative to Kafka MirrorMaker 2. - [Retrieval-Augmented Generation (RAG)](rag/) How to configure Redpanda Connect to create a RAG pipeline, using PostgreSQL and PGVector. - [Custom Metrics](custom_metrics/) Learn how to emit custom metrics from messages. - [Create a Discord Bot](discord_bot/) Learn how to use Redpanda Connect to create a Discord bot. --- # Page 347: CDC Schema Registry Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/cdc_schema_registry.md --- # CDC Schema Registry Patterns --- title: CDC Schema Registry Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: cdc_schema_registry page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cdc_schema_registry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/cdc_schema_registry.adoc description: Register schemas from database change data capture (CDC) events in a schema registry and stream schema-encoded messages to Redpanda with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply patterns that register CDC table schemas in a schema registry automatically learning-objective-2: Adapt the patterns to PostgreSQL, MySQL, SQL Server, and Oracle CDC inputs learning-objective-3: Identify the data types and transformations that prevent a message from encoding page-git-created-date: "2026-10-14" page-git-modified-date: "2026-10-14" --- Relational [CDC inputs](https://docs.redpanda.com/connect/guides/cdc/) attach each table’s schema to every message they emit. The [`schema_registry_encode`](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/) processor reads that schema, registers it in a schema registry, and encodes the message with it, so you can stream schema-encoded change events to Redpanda without registering schemas ahead of time. Use these patterns when downstream consumers expect schema-registry-encoded records from your database tables. Use this cookbook to: - Apply patterns that register CDC table schemas in a schema registry automatically - Adapt the patterns to PostgreSQL, MySQL, SQL Server, and Oracle CDC inputs - Identify the data types and transformations that prevent a message from encoding ## [](#how-it-works)How it works Each CDC input adds a `schema` metadata field to the messages it emits. The field holds the source table’s columns and types in a common schema format, and a fingerprint that identifies that version of the table structure. When you set `schema_metadata` on the `schema_registry_encode` processor, the processor uses that metadata instead of fetching a schema from the registry. For each message, the processor: 1. Reads the schema from the metadata key named in `schema_metadata`. 2. Converts the schema to the format set in `format`. 3. Registers the schema under the subject set in `subject`, unless it has already registered that schema version for that subject. 4. Encodes the message with the schema and prefixes it with the schema ID, in the standard schema registry wire format. When a table’s structure changes, the fingerprint changes and the processor registers a new schema version under the same subject. The registry’s compatibility rules for that subject decide whether it accepts the new version. If the registry rejects it, the message fails to encode. The following inputs emit `schema` metadata: | Input | Available from | Table name metadata | | --- | --- | --- | | postgres_cdc | 4.82.0 | table | | mysql_cdc | 4.81.0 | table | | microsoft_sql_server_cdc | 4.83.0 | table | | oracledb_cdc | 4.84.0 | table_name | The `schema_metadata` field of the `schema_registry_encode` processor requires Redpanda Connect 4.83.0 or later. ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [rpk installation](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#enterprise-license)Enterprise license The CDC inputs in this cookbook require a Redpanda Enterprise Edition license. See [Enterprise Licensing](https://docs.redpanda.com/connect/get-started/licensing/). ### [](#source-database-and-schema-registry)Source database and schema registry Configure the source database for CDC as described in the cookbook for your database: - [PostgreSQL CDC Patterns](https://docs.redpanda.com/connect/cookbooks/postgres_cdc/) - [MySQL CDC Patterns](https://docs.redpanda.com/connect/cookbooks/mysql_cdc/) - [SQL Server CDC Patterns](https://docs.redpanda.com/connect/cookbooks/microsoft_sql_server_cdc/) - [Oracle Database CDC Patterns](https://docs.redpanda.com/connect/cookbooks/oracledb_cdc/) You also need a schema registry that the pipeline can write to, such as Redpanda Schema Registry. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export SCHEMA_REGISTRY_URL=http://localhost:8081 (1) export REDPANDA_BROKERS=localhost:9092 (2) ``` | 1 | The schema registry URL. | | --- | --- | | 2 | The Redpanda broker addresses. | Each example also uses the connection variables from the cookbook for its database, such as `PG_DSN`, `PG_SCHEMA`, and `PG_SLOT_NAME` for PostgreSQL. ## [](#register-avro-schemas-from-postgresql)Register Avro schemas from PostgreSQL Encode PostgreSQL change events as Avro and write each table to its own topic: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders - customers slot_name: ${PG_SLOT_NAME} stream_snapshot: true pipeline: processors: - schema_registry_encode: url: ${SCHEMA_REGISTRY_URL} subject: ${! meta("table") }-value schema_metadata: schema format: avro avro: raw_json: true output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("table") } ``` This pattern: - Reads the table schema from the `schema` metadata field and converts it to an Avro schema. - Registers the schema under a subject named after the table, such as `orders-value`. Because the output writes each table to a topic with the same name, the subject follows the `-value` naming convention that most schema registry clients expect. - Sets `avro.raw_json` to `true`, because CDC inputs emit standard JSON rather than Avro JSON. In `schema_metadata` mode with the `avro` format, the processor refuses to start unless you set `avro.raw_json` explicitly. ## [](#register-avro-schemas-from-mysql)Register Avro schemas from MySQL The same processor configuration works with the `mysql_cdc` input: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders - customers checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - schema_registry_encode: url: ${SCHEMA_REGISTRY_URL} subject: ${! meta("table") }-value schema_metadata: schema format: avro avro: raw_json: true output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("table") } cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` The `mysql_cdc` input also needs a checkpoint cache, configured here as a Redis cache resource. ## [](#register-avro-schemas-from-sql-server)Register Avro schemas from SQL Server The `microsoft_sql_server_cdc` input also stores the table name in the `table` metadata field, so the PostgreSQL processor and output configuration works unchanged. Replace the input: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders - dbo.customers stream_snapshot: true ``` By default, the `microsoft_sql_server_cdc` input stores its checkpoint in a table in the source database, so it doesn’t need a cache resource. ## [](#register-avro-schemas-from-oracle)Register Avro schemas from Oracle The `oracledb_cdc` input stores the table name in the `table_name` metadata field instead of `table`. Oracle also returns unquoted table names in uppercase, so this pattern lowercases the name before using it for the topic and subject: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: - mapping: | meta topic = meta("table_name").lowercase() - schema_registry_encode: url: ${SCHEMA_REGISTRY_URL} subject: ${! meta("topic") }-value schema_metadata: schema format: avro avro: raw_json: true output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } ``` The `mapping` processor changes only metadata, so the message body and its `schema` metadata reach `schema_registry_encode` unchanged. ## [](#filter-events-before-encoding)Filter events before encoding Drop events you don’t need and set the message key before encoding: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders slot_name: ${PG_SLOT_NAME} stream_snapshot: true pipeline: processors: - mapping: | root = if meta("operation") == "delete" { deleted() } meta key = this.id.string() - schema_registry_encode: url: ${SCHEMA_REGISTRY_URL} subject: orders-value schema_metadata: schema format: avro avro: raw_json: true output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: orders key: ${! meta("key") } ``` This pattern: - Drops `delete` events, passing through snapshot (`read`), `insert`, and `update` events. - Copies the primary key into the `key` metadata field before encoding. After `schema_registry_encode` runs, the message body is binary Avro, so the output can’t read fields from it with `json()`. Update `this.id` to match your table’s primary key column. - Uses a fixed subject because the pipeline reads a single table. Processors that run before `schema_registry_encode` keep the `schema` metadata field. Don’t remove metadata before the encoder, for example with `meta = deleted()`, or messages fail with an error that the schema metadata key wasn’t found. ## [](#consume-schema-encoded-events)Consume schema-encoded events Read the encoded events from Redpanda and decode them back to JSON with the [`schema_registry_decode`](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) processor: ```yaml input: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topics: - orders consumer_group: orders-reader pipeline: processors: - schema_registry_decode: url: ${SCHEMA_REGISTRY_URL} avro: raw_unions: true output: stdout: codec: lines ``` The decoder reads the schema ID from each message and fetches the matching schema from the registry, so it doesn’t need a subject. The generated Avro schemas make every column nullable, so each value is an Avro union. Setting `avro.raw_unions` to `true` decodes values as plain JSON, such as `"id":1`, instead of Avro JSON, such as `"id":{"int":1}`. ## [](#troubleshoot-encoding-errors)Troubleshoot encoding errors When a message fails to encode, `schema_registry_encode` leaves it unchanged and flags it with an error. If you don’t handle the error, the output writes the unencoded JSON message to the topic. Handle these messages with the [error handling](https://docs.redpanda.com/connect/configuration/error_handling/) processors, for example by routing them to a dead-letter topic. ### [](#message-body-doesnt-match-the-table-schema)Message body doesn’t match the table schema The registered schema describes the source table’s columns. If a processor reshapes the message body before `schema_registry_encode`, the body no longer matches the schema: - Avro encoding drops fields that aren’t in the schema. Versions earlier than 4.84.1 fail to encode the message instead. - Avro encoding writes `null` for a column that’s missing from the body, because the generated schemas make every column optional. The message encodes without an error, so a column that a processor drops by mistake reaches the topic as `null`. To publish a different shape, transform the message after it’s decoded downstream, or use `schema_registry_encode` without `schema_metadata` and a schema you register yourself. ### [](#json-columns)JSON columns The generated Avro schema types PostgreSQL `json` and `jsonb` columns, MySQL `JSON` columns, and PostgreSQL column types that have no schema mapping as `bytes`. The CDC inputs decode these columns into JSON objects, so the `avro` format fails to encode any row where the column holds a value: ```text avro: field attrs: cannot use map[string]any with Avro type bytes ``` Rows where the column is `null` encode without an error. To encode these rows, convert the column to a JSON string with a [`mutation`](https://docs.redpanda.com/connect/components/processors/mutation/) processor before `schema_registry_encode`. This example converts a column named `attrs`: ```yaml - mutation: | root.attrs = if this.attrs != null { this.attrs.format_json(no_indent: true).string() } ``` Use `mutation` rather than `mapping`, so the rest of the message body and its metadata stay unchanged. The `.string()` call matters: `format_json()` returns bytes, which the message body would carry as a base64 string. Downstream consumers receive the column as a JSON string and parse it after decoding. ### [](#unconstrained-numeric-columns)Unconstrained numeric columns Avro decimals need a fixed precision and scale. The following column types have neither, so the `avro` format fails to convert schemas that contain them: - PostgreSQL `NUMERIC` declared without precision and scale. - Oracle `NUMBER` declared without precision, `FLOAT`, and `NUMBER` columns whose scale is greater than the precision or negative. Declare the column with a precision and scale, such as `NUMERIC(10,2)` or `NUMBER(10,2)`. ### [](#transaction-markers)Transaction markers When you enable `include_transaction_markers` on the `postgres_cdc` input, `begin` and `commit` messages carry no `schema` metadata, so they fail to encode. Drop them before `schema_registry_encode`: ```yaml - mapping: | root = if meta("operation") == "begin" || meta("operation") == "commit" { deleted() } ``` ### [](#json-schema-format)JSON Schema format The `format` field also accepts `json_schema`. In this format, the processor registers a JSON Schema and validates each message against it instead of encoding to binary. The generated JSON Schema treats nullable columns as optional properties but doesn’t accept `null` as a value, so a row that contains a `NULL` column fails validation. Use the `avro` format for tables with nullable columns. ## [](#next-steps)Next steps - [`schema_registry_encode` processor reference](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/) - [`schema_registry_decode` processor reference](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) - [Change Data Capture (CDC)](https://docs.redpanda.com/connect/guides/cdc/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 348: Custom Metrics **URL**: https://docs.redpanda.com/connect/cookbooks/custom_metrics.md --- # Custom Metrics --- title: Custom Metrics latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: custom_metrics page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: custom_metrics.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/custom_metrics.adoc description: Learn how to emit custom metrics from messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-31" --- You can’t build graphs without metrics, and [Redpanda Connect emits many](https://docs.redpanda.com/connect/components/metrics/about/). However, occasionally you might want to also emit custom metrics that track data extracted from messages being processed. In this cookbook we’ll explore how to achieve this by configuring Redpanda Connect to pull download stats from Github, DockerHub and Homebrew and emit them as gauges. ## [](#the-basics)The basics Firstly, we need to target an API so let’s start with the nice and simple Homebrew API, which we’ll poll every 60 seconds. We can either do it with an [`http_client` input](https://docs.redpanda.com/connect/components/inputs/http_client/) and a [rate limit](https://docs.redpanda.com/connect/components/rate_limits/about/) that restricts us to one request per 60 seconds, or we can use a [`generate` input](https://docs.redpanda.com/connect/components/inputs/generate/) to generate a message every 60 seconds that triggers an [`http` processor](https://docs.redpanda.com/connect/components/processors/http/): ### Processor ```yaml input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET ``` ### Input ```yaml input: http_client: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET rate_limit: brewlimit rate_limit_resources: - label: brewlimit local: count: 1 interval: 60s ``` For this cookbook we’ll continue with the processor option as it makes it easier to deploy it as a [scheduled lambda function](https://docs.redpanda.com/connect/guides/serverless/lambda/) later on, which is how I’m currently doing it in real life. The homebrew formula API gives us a JSON blob that looks like this (removing fields we’re not interested in, and with numbers inflated relative to my ego): ```json { "name":"benthos", "desc":"Stream processor for mundane tasks written in Go", "analytics":{"install":{"30d":{"benthos":78978979},"90d":{"benthos":253339124},"365d":{"benthos":681356871}}} } ``` This format makes it fairly easy to emit the value of `analytics.install.30d.benthos` as a gauge with the [`metric` processor](https://docs.redpanda.com/connect/components/processors/metric/): ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET - metric: type: gauge name: downloads labels: source: homebrew value: ${! json("analytics.install.30d.benthos") } - mapping: root = deleted() metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` With the above config we have selected the [`prometheus` metrics type](https://docs.redpanda.com/connect/components/metrics/prometheus/), which allows us to use [Prometheus](https://prometheus.io/) to scrape metrics from Redpanda Connect by polling its HTTP API at the url `http://localhost:4195/stats`. We have also specified a \[`path_mapping`\]\[metrics.prometheus.path\_mapping\] that deletes any internal metrics usually emitted by Redpanda Connect by filtering on our custom metric name. Finally, there’s also a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) added to the end of our pipeline that deletes all messages since we’re not interested in sending the raw data anywhere after this point anyway. While running this config you can verify that our custom metric is emitted with `curl`: ```sh curl -s http://localhost:4195/stats | grep downloads ``` Giving something like: ```text # HELP benthos_downloads Redpanda Connect Gauge metric # TYPE benthos_downloads gauge benthos_downloads{source="homebrew"} 78978979 ``` Easy! The DockerHub API is also pretty simple, and adding it to our pipeline is just: ### Diff ```diff source: homebrew value: ${! json("analytics.install.30d.benthos") } + - mapping: root = "" + + - http: + url: http://docker.redpanda.com/redpandadata/connect + verb: GET + headers: + Content-Type: application/json + + - metric: + type: gauge + name: downloads + labels: + source: dockerhub + value: ${! json("pull_count") } + - mapping: root = deleted() ``` ### Full Config ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET - metric: type: gauge name: downloads labels: source: homebrew value: ${! json("analytics.install.30d.benthos") } - mapping: root = "" - http: url: http://docker.redpanda.com/redpandadata/connect verb: GET headers: Content-Type: application/json - metric: type: gauge name: downloads labels: source: dockerhub value: ${! json("pull_count") } - mapping: root = deleted() metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` ## [](#advanced-custom-metrics-example)Advanced custom metrics example So that’s the basics covered. Next, we’re going to target the Github releases API which gives a slightly more complex payload that looks something like this: ```json [ { "tag_name": "X.XX.X", "assets":[ {"name":"benthos-lambda_X.XX.X_linux_amd64.zip","download_count":543534545}, {"name":"benthos_X.XX.X_darwin_amd64.tar.gz","download_count":43242342}, {"name":"benthos_X.XX.X_freebsd_amd64.tar.gz","download_count":534565656}, {"name":"benthos_X.XX.X_linux_amd64.tar.gz","download_count":743282474324} ] } ] ``` It’s an array of objects, one for each tagged release, with a field `assets` which is an array of objects representing each release asset, of which we want to emit a separate download gauge. In order to do this we’re going to use a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) to remap the payload from Github into an array of objects of the following form: ```json [ {"source":"github","dist":"lambda_linux_amd64","download_count":543534545,"version":"X.XX.X"}, {"source":"github","dist":"darwin_amd64","download_count":43242342,"version":"X.XX.X"}, {"source":"github","dist":"freebsd_amd64","download_count":534565656,"version":"X.XX.X"}, {"source":"github","dist":"linux_amd64","download_count":743282474324,"version":"X.XX.X"} ] ``` Then we can use an [`unarchive` processor](https://docs.redpanda.com/connect/components/processors/unarchive/) with the format `json_array` to expand this array into N individual messages, one for each asset. Finally, we will follow up with a [`metric` processor](https://docs.redpanda.com/connect/components/processors/metric/) that dynamically sets labels following the fields `source`, `dist` and `version` so that we have a separate metrics series for each asset type for each tagged version. A simple pipeline of these steps would look like this (please forgive the regexp): ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://api.github.com/repos/redpanda-data/connect/releases verb: GET - mapping: | root = this.map_each(release -> release.assets.map_each(asset -> { "source": "github", "dist": asset.name.re_replace_all("^benthos-?((lambda_)|_)[0-9\\.]+(-rc[0-9]+)?_([^\\.]+).*", "$2$4"), "download_count": asset.download_count, "version": release.tag_name.trim("v"), }).filter(asset -> asset.dist != "checksums")).flatten() - unarchive: format: json_array - metric: type: gauge name: downloads labels: dist: ${! json("dist") } source: ${! json("source") } value: ${! json("download_count") } - mapping: root = deleted() metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` Finally, let’s combine all the custom metrics into one pipeline. ## [](#combining-into-a-workflow)Combining into a workflow The following config expands on the previous examples by configuring each API poll as a [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/), which allows us to run them within a [`workflow` processor](https://docs.redpanda.com/connect/components/processors/workflow/) that can execute all three branches in parallel. The [`metric` processors](https://docs.redpanda.com/connect/components/processors/metric/) have also been combined into a single reusable resource by updating the other API calls to format their payloads into the same structure as our Github remap. ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = {} pipeline: processors: - workflow: meta_path: results order: [ [ dockerhub, github, homebrew ] ] processor_resources: - label: dockerhub branch: request_map: 'root = ""' processors: - try: - http: url: http://docker.redpanda.com/redpandadata/connect verb: GET headers: Content-Type: application/json - mapping: | root.source = "docker" root.dist = "docker" root.download_count = this.pull_count root.version = "all" - resource: metric_gauge - label: github branch: request_map: 'root = ""' processors: - try: - http: url: https://api.github.com/repos/redpanda-data/connect/releases verb: GET - mapping: | root = this.map_each(release -> release.assets.map_each(asset -> { "source": "github", "dist": asset.name.re_replace_all("^benthos-?((lambda_)|_)[0-9\\.]+(-rc[0-9]+)?_([^\\.]+).*", "$2$4"), "download_count": asset.download_count, "version": release.tag_name.trim("v"), }).filter(asset -> asset.dist != "checksums")).flatten() - unarchive: format: json_array - resource: metric_gauge - mapping: 'root = if batch_index() != 0 { deleted() }' - label: homebrew branch: request_map: 'root = ""' processors: - try: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET - mapping: | root.source = "homebrew" root.dist = "homebrew" root.download_count = this.analytics.install.30d.benthos root.version = "all" - resource: metric_gauge - label: metric_gauge metric: type: gauge name: downloads labels: dist: ${! json("dist") } source: ${! json("source") } version: ${! json("version") } value: ${! json("download_count") } metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` --- # Page 349: Create a Discord Bot **URL**: https://docs.redpanda.com/connect/cookbooks/discord_bot.md --- # Create a Discord Bot --- title: Create a Discord Bot latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: discord_bot page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: discord_bot.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/discord_bot.adoc description: Learn how to use Redpanda Connect to create a Discord bot. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-31" --- This cookbook outlines how Redpanda Connect can be used to create a Discord bot for important tasks such as providing insults and bad jokes to your chat. If you’re a member of the [Redpanda Connect Discord server](https://discord.gg/6VaWjzP) then you’re likely already familiar with Blob Bot which is the resulting product. ## [](#consuming-messages)Consuming messages Before you start messing with Redpanda Connect you need to register a new bot with the [Discord Developer Portal](https://discord.com/developers/applications). Start by building an Application, then use the build-a-bot page to choose a bot name and avatar. You should end up with a token generated for the bot, and you’ll also need to add it to your server. As soon as your bot is added to your server and you have a token you can immediately begin consuming messages from a channel with the [`discord` input](https://docs.redpanda.com/connect/components/inputs/discord/): ```yaml input: discord: channel_id: ${DISCORD_CHANNEL} bot_token: ${DISCORD_BOT_TOKEN} cache: request_tracking cache_resources: - label: request_tracking file: directory: /tmp/discord_bot ``` > If you aren’t sure how to access the ID of a channel try [this tutorial](https://support.discord.com/hc/en-us/articles/206346498-Where-can-I-find-my-User-Server-Message-ID-). The `poll_period` shouldn’t be too short as it’ll exhaust your rate limits. If you plan to use the bot for hitting multiple Discord APIs then give it a fair few seconds between each poll. It’s also necessary to point the input to a [cache resource](https://docs.redpanda.com/connect/components/caches/about/), and this will be used to store the ID of the latest message received for paginating the messages endpoint. The `limit` is the maximum number of messages to consume from the channel when we haven’t got a message to track and are consuming a backlog. The first time we run our bot we will pull a maximum of 10 of the latest messages in the channel, the maximum you can set here is 100. If you were to run this config (setting the channel and bot token as env vars in a file called `testing.env`) you’ll see it print messages from the channel to stdout in JSON form: ```bash rpk connect run -e testing.env ./config.yaml {"content":"so i like totally just tripped over my own network cables","author":{"id":"1234"}} {"content":"like omg that is SO you!!!","author":{"id":"4321"}} {"content":"yas totally","author":{"id":"1234"}} {"content":"yeah","author":{"id":"4321"}} ``` ## [](#writing-messages)Writing messages Writing messages to a Discord channel is pretty easy. You can feed the [`discord` output](https://docs.redpanda.com/connect/components/outputs/discord/) either a JSON object following the [Message Object structure](https://discord.com/developers/docs/resources/channel#message-object), or just a raw string and the structure will be created for you. Therefore we can write a hypothetical uppercasing echo bot with a simple [Bloblang mapping](https://docs.redpanda.com/connect/guides/bloblang/about/): ```yaml pipeline: processors: - mapping: | root = if !this.content.has_prefix("SHOUTS BACK") { "SHOUTS BACK BOT SAYS " + this.content.uppercase() } else { deleted() } output: discord: channel_id: ${DISCORD_CHANNEL} bot_token: ${DISCORD_BOT_TOKEN} ``` If we add that to the end of the first config you should see the bot respond to messages in the channel by posting an uppercase version of it with a prefix. Note that we also delete the message in our mapping if it has the same prefix that we’re adding ourselves, which is a quick and dirty way of ensuring the bot doesn’t echo its own messages. ## [](#custom-commands)Custom commands Shout bot is clearly an absolute riot and a true fan favourite. However, it will get old fast. Let’s make our bot more elegant by introducing some commands by swapping our plain mapping with a [`switch` processor](https://docs.redpanda.com/connect/components/processors/switch/): ```yaml pipeline: processors: - switch: - check: this.type == 7 processors: - mapping: 'root = "Welcome to the server <@%v>!".format(this.author.id)' - processors: - mapping: 'root = deleted()' ``` By changing our mapping out to this switch we can add specialised commands for different message types, and if none of the cases match then we don’t respond. Technically, we can do all of this within a single Bloblang mapping by using a match expression, but having a switch processor would also allow us to add cases where we do cool things like hit other APIs, etc. The only case we’ve added here is one that activates when the message type is a specific one sent when a new person joins, and in response we give them a warm welcome. The welcome mentions the new user by injecting the user id into the welcome string with `.format(this.author.id)`, which replaces the `%v` placeholder with the author ID (the user that joined and therefore created the join message). This response is good but not very interactive. Let’s add a few commands that people can play with: ```yaml pipeline: processors: - switch: - check: this.type == 7 processors: - mapping: 'root = "Welcome to the server <@%v>!".format(this.author.id)' - check: this.content == "/joke" processors: - mapping: | let jokes = [ "What do you call a belt made of watches? A waist of time.", "What does a clock do when it’s hungry? It goes back four seconds.", "A company is making glass coffins. Whether they’re successful remains to be seen.", ] root = $jokes.index(timestamp_unix_nano() % $jokes.length()) - check: this.content == "/roast" processors: - mapping: | let roasts = [ "If <@%v>'s brain was dynamite, there wouldn’t be enough to blow their hat off.", "Someday you’ll go far <@%v>, and I really hope you stay there.", "I’d give you a nasty look, but you’ve already got one <@%v>.", ] root = $roasts.index(timestamp_unix_nano() % $roasts.length()).format(this.author.id) - processors: - mapping: 'root = deleted()' ``` Here we have two new commands. If someone posts a message "/joke" then we respond by selecting one of several exceptionally funny jokes from a static list in the mapping. The second new command is "/roast" and is exclusively for brave souls as the responses can be cruel and personal. The command works similarly to "/joke" with the exception being the ID of the user that made the command will be injected into the roast, as mentioning the target of the roast makes it significantly more heartbreaking (as intended). ## [](#hitting-other-apis)Hitting other APIs Clicking websites and browsing the internet is very difficult and most people are simply too busy for it, it’d therefore be useful if we could have our bot do some browsing for us occasionally. The final command we’re going to add to our bot is "/release", where it will hit the Github API and find out for us what the latest Redpanda Connect release is: ```yaml pipeline: processors: - switch: # Other cases omitted for brevity - check: this.content == "/release" processors: - mapping: 'root = ""' - try: - http: url: https://api.github.com/repos/redpanda-data/connect/releases/latest verb: GET - mapping: 'root = "The latest release of Redpanda Connect is %v: %v".format(this.tag_name, this.html_url)' - catch: - log: fields_mapping: 'root.error = error()' message: "Failed to process message" - mapping: 'root = "Sorry, my circuits are all bent from twerking and I must have malfunctioned."' ``` Here we’ve added a switch case that clears the contents of the message, hits the Github API to obtain the latest Redpanda Connect release as a JSON object, and finally maps the tag name and the URL of the release to a useful message. > We’re hitting the Github API with the [generic `http` processor](https://docs.redpanda.com/connect/components/processors/http/), which can be configured to work with most HTTP based APIs. In fact, the Discord input and output are actually [configuration templates](https://docs.redpanda.com/connect/configuration/templating/) that use the generic HTTP components [under the hood](https://github.com/redpanda-data/connect/blob/master/template/outputs/discord.yaml). Since this command is networked and therefore has a chance of failure we’ve added some [error handling](https://docs.redpanda.com/connect/configuration/error_handling/) mechanisms after the switch processor so that it’d capture errors from this new case and any new cases we add later. Within the catch block we simply log the error for the admin to peruse and change the response message out for a generic "whoopsie daisy" apology. The full config for Blob Bot (with some super secret responses redacted) can be found [in the Github repo](https://github.com/redpanda-data/connect/blob/master/config/examples/discord_bot.yaml). To find out more about Bloblang check out [the guide page](https://docs.redpanda.com/connect/guides/bloblang/about/). To find out more about config templates check out the [templates documentation page](https://docs.redpanda.com/connect/configuration/templating/). If you want to play with Blob Bot then [join our Discord](https://discord.gg/6VaWjzP). There are also some humans in there that will help you manage your disappointment when you see Blob Bot in action. --- # Page 350: DynamoDB CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/dynamodb_cdc.md --- # DynamoDB CDC Patterns --- title: DynamoDB CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: dynamodb_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: dynamodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/dynamodb_cdc.adoc description: Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing DynamoDB CDC events learning-objective-2: Adapt integration patterns to route CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-03-02" page-git-modified-date: "2026-10-14" --- The DynamoDB CDC input enables capturing item-level changes from DynamoDB tables with streams enabled. This cookbook provides reusable patterns for filtering, transforming, and routing DynamoDB CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing DynamoDB CDC events - Adapt integration patterns to route CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#dynamodb-streams)DynamoDB Streams The source DynamoDB table must have [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) enabled with an appropriate view type: - `KEYS_ONLY`: Only the key attributes of the modified item - `NEW_IMAGE`: The entire item as it appears after the modification - `OLD_IMAGE`: The entire item as it appeared before the modification - `NEW_AND_OLD_IMAGES`: Both the new and old item images (recommended for detecting changes) To enable streams on an existing table using the AWS CLI: ```bash aws dynamodb update-table \ --table-name orders \ --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES ``` ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for AWS configuration. Environment variables keep credentials out of your pipeline configuration files. ```bash export DYNAMODB_TABLE=orders (1) export AWS_REGION=us-east-1 (2) export REDPANDA_BROKERS=localhost:9092 (3) export S3_BUCKET=cdc-archive (4) ``` | 1 | The name of the DynamoDB table with streams enabled. | | --- | --- | | 2 | The AWS region where your DynamoDB table is located. | | 3 | The Redpanda broker addresses (for Redpanda output examples). | | 4 | The S3 bucket name (for S3 output examples). | Redpanda Connect loads AWS credentials from the standard [credential chain](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (environment variables, `~/.aws/credentials`, or IAM roles). ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from a DynamoDB table and outputs them with metadata: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon pipeline: processors: # Extract the change event details - mapping: | root.event_type = this.eventName root.table = this.tableName root.event_id = this.eventID root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage root.sequence_number = this.dynamodb.sequenceNumber root.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available fields for Bloblang mappings, see the [message structure](https://docs.redpanda.com/connect/components/inputs/aws_dynamodb_cdc/#message-structure) section in the connector reference. ## [](#filter-cdc-events)Filter CDC events Filter events to process only specific change types: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} start_from: latest pipeline: processors: # Filter to only process INSERT and MODIFY events (ignore REMOVE) - mapping: | root = if this.eventName == "REMOVE" { deleted() } else { this } # Transform to a simplified format - mapping: | root.event_type = this.eventName root.keys = this.dynamodb.keys root.new_data = this.dynamodb.newImage root.old_data = this.dynamodb.oldImage output: stdout: codec: lines ``` This example: - Filters out `REMOVE` events using `deleted()` - Transforms the event to a simplified format ## [](#route-to-redpanda)Route to Redpanda Stream DynamoDB changes to Redpanda for real-time processing: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints batch_size: 100 poll_interval: 500ms pipeline: processors: # Transform to a Kafka-friendly format with a composite key - mapping: | let keys = this.dynamodb.keys meta kafka_key = [$keys.pk, $keys.sk].filter(v -> v != null).join("#") root.event_type = this.eventName root.table = this.tableName root.timestamp = now() root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-cdc-events key: ${! @kafka_key } partitioner: murmur2_hash compression: snappy batching: count: 100 period: 1s ``` This example: - Creates a composite message key from the DynamoDB primary key - Transforms the DynamoDB format to plain JSON - Batches messages for efficient delivery ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon pipeline: processors: # Add partitioning metadata for S3 organization - mapping: | let event_time = now() meta s3_path = "year=%s/month=%s/day=%s/hour=%s".format( $event_time.ts_format("2006"), $event_time.ts_format("01"), $event_time.ts_format("02"), $event_time.ts_format("15") ) root.event_type = this.eventName root.table = this.tableName root.sequence_number = this.dynamodb.sequenceNumber root.event_time = $event_time root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage output: aws_s3: bucket: ${S3_BUCKET} path: dynamodb-cdc/${DYNAMODB_TABLE}/${! @s3_path }/${! uuid_v4() }.json region: ${AWS_REGION} object_canned_acl: private batching: count: 1000 period: 1m processors: - archive: format: lines ``` This example: - Organizes files by time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} pipeline: processors: # Transform to a common format - mapping: | root.event_type = this.eventName root.table = this.tableName root.timestamp = now() root.keys = this.dynamodb.keys root.data = if this.dynamodb.exists("newImage") { this.dynamodb.newImage } else { this.dynamodb.oldImage } output: switch: cases: # Route INSERT events to a topic for new records - check: this.event_type == "INSERT" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-inserts # Route MODIFY events to a topic for updates - check: this.event_type == "MODIFY" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-updates # Route REMOVE events to a topic for deletes - check: this.event_type == "REMOVE" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-deletes # Fallback for any unexpected event types - output: drop: {} ``` This pattern: - Separates processing pipelines for inserts, updates, and deletes - Applies different retention policies per event type - Supports specialized downstream consumers ## [](#detect-changed-fields)Detect changed fields Compare old and new images to identify which fields changed: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} pipeline: processors: # Only process MODIFY events - mapping: | root = if this.eventName != "MODIFY" { deleted() } else { this } # Compare old and new images to find changed fields - mapping: | let old_data = this.dynamodb.oldImage let new_data = this.dynamodb.newImage root.table = this.tableName root.keys = this.dynamodb.keys root.timestamp = now() # Find fields that changed by comparing key-value pairs root.changes = $new_data.key_values().filter(kv -> !$old_data.exists(kv.key) || $old_data.get(kv.key) != kv.value).map_each(kv -> {"field": kv.key, "old_value": if $old_data.exists(kv.key) { $old_data.get(kv.key) } else { null }, "new_value": kv.value}) # Find fields that were removed root.removed_fields = $old_data.keys().filter(k -> !$new_data.exists(k)) output: stdout: codec: lines ``` This pattern: - Filters to only MODIFY events - Compares old and new images to find differences - Outputs a list of changed fields with their old and new values > 📝 **NOTE** > > This pattern requires the `NEW_AND_OLD_IMAGES` stream view type. The `.key_values()` method converts an object to an array of key-value pairs that can be filtered and mapped. ## [](#checkpointing)Checkpointing The DynamoDB CDC input automatically manages checkpoints in a separate DynamoDB table: ```yaml input: aws_dynamodb_cdc: tables: - orders checkpoint_table: cdc-checkpoints (1) checkpoint_limit: 500 (2) start_from: trim_horizon (3) ``` | 1 | Custom checkpoint table name (default: redpanda_dynamodb_checkpoints). | | --- | --- | | 2 | Checkpoint after every 500 messages (lower = better recovery, higher = fewer writes). | | 3 | Start from the oldest available record when no checkpoint exists. | If a checkpoint table doesn’t exist, it’s created automatically with the required schema. ## [](#performance-tuning)Performance tuning Optimize throughput and latency with these settings: ```yaml input: aws_dynamodb_cdc: tables: - orders batch_size: 1000 (1) poll_interval: 100ms (2) max_tracked_shards: 10000 (3) throttle_backoff: 50ms (4) ``` | 1 | Maximum records per shard per request (1-1000). | | --- | --- | | 2 | Time between polls when no records are available. | | 3 | Maximum shards to track (for very large tables). | | 4 | Backpressure delay when too many messages are in-flight. | ### [](#throughput-considerations)Throughput considerations - DynamoDB Streams allows 5 `GetRecords` calls per second per shard - Higher `batch_size` improves throughput but increases memory usage - Shorter `poll_interval` reduces latency but increases API calls ## [](#troubleshoot-common-issues)Troubleshoot common issues ### [](#no-events-received)No events received If you’re not receiving events: 1. Verify streams are enabled on the table: ```bash aws dynamodb describe-table --table-name orders \ --query 'Table.StreamSpecification' ``` 2. Check that changes are being made to the table 3. Verify `start_from` is set to `trim_horizon` to capture existing stream data ### [](#duplicate-events)Duplicate events Each stream record appears exactly once in DynamoDB Streams. However, if your pipeline fails before checkpointing, records may be re-read on restart, resulting in at-least-once processing semantics. To handle potential duplicates: - Use idempotent processing in downstream systems - Deduplicate using the `dynamodb_sequence_number` metadata - Lower `checkpoint_limit` to reduce the window of possible duplicates ### [](#stream-retention)Stream retention DynamoDB Streams retains data for 24 hours. If your pipeline is offline longer than that: - Consider using [Kinesis Data Streams for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html) with the [`aws_kinesis` input](https://docs.redpanda.com/connect/components/inputs/aws_kinesis/) instead (up to 1 year retention) - Implement a full-table scan fallback for disaster recovery ## [](#next-steps)Next steps - [DynamoDB CDC Input Reference](https://docs.redpanda.com/connect/components/inputs/aws_dynamodb_cdc/) - [AWS Configuration Guide](https://docs.redpanda.com/connect/guides/cloud/aws/) - [Kinesis Input](https://docs.redpanda.com/connect/components/inputs/aws_kinesis/) (for Kinesis Data Streams for DynamoDB) - [DynamoDB Streams Documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 351: Enrichment Workflows **URL**: https://docs.redpanda.com/connect/cookbooks/enrichments.md --- # Enrichment Workflows --- title: Enrichment Workflows latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: enrichments page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: enrichments.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/enrichments.adoc description: How to configure Redpanda Connect to process a workflow of enrichment services. page-git-created-date: "2024-05-24" page-git-modified-date: "2025-04-10" --- This cookbook demonstrates how to enrich a stream of JSON documents with HTTP services. This method also works with [AWS Lambda functions](https://docs.redpanda.com/connect/components/processors/aws_lambda/), [subprocesses](https://docs.redpanda.com/connect/components/processors/subprocess/), etc. We will start off by configuring a single enrichment, then we will move onto a workflow of enrichments with a network of dependencies using the [`workflow` processor](https://docs.redpanda.com/connect/components/processors/workflow/). Each enrichment will be performed in parallel across a [pre-batched](https://docs.redpanda.com/connect/configuration/batching/) stream of documents. Workflow enrichments that do not depend on each other will also be performed in parallel, making this orchestration method very efficient. The imaginary problem we are going to solve is applying a set of NLP based enrichments to a feed of articles in order to detect fake news. We will be consuming and writing to Kafka, but the example works with any [input](https://docs.redpanda.com/connect/components/inputs/about/) and [output](https://docs.redpanda.com/connect/components/outputs/about/) combination. Articles are received over the topic `articles` and look like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } } ``` ## [](#meet-the-enrichments)Meet the enrichments ### [](#claims-detector)Claims detector To start us off we will configure a single enrichment, which is an imaginary 'claims detector' service. This is an HTTP service that wraps a trained machine learning model to extract claims that are made within a body of text. The service expects a `POST` request with JSON payload of the form: ```json { "text": "The world was shocked this morning to find that all dogs have stopped barking." } ``` And returns a JSON payload of the form: ```json { "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } ``` Since each request only applies to a single document we will make this enrichment scale by deploying multiple HTTP services and hitting those instances in parallel across our document batches. In order to send a mapped request and map the response back into the original document we will use the [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/), with a child [`http`](https://docs.redpanda.com/connect/components/processors/http/) processor. ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group batching: count: 20 # Tune this to set the size of our document batches. period: 1s pipeline: processors: - branch: request_map: 'root.text = this.article.content' processors: - http: url: http://localhost:4197/claims verb: POST result_map: 'root.tmp.claims = this.claims' output: kafka: addresses: [ TODO ] topic: comments_hydrated ``` With this pipeline our documents will come out looking something like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." }, "tmp": { "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } } ``` ### [](#hyperbole-detector)Hyperbole detector Next up is a 'hyperbole detector' that takes a `POST` request containing the article contents and returns a hyperbole score between 0 and 1. This time the format is array-based and therefore supports calculating multiple documents in a single request, making better use of the host machines GPU. A request should take the following form: ```json [ { "text": "The world was shocked this morning to find that all dogs have stopped barking." } ] ``` And the response looks like this: ```json [ { "hyperbole_rank": 0.73 } ] ``` In order to create a single request from a batch of documents, and subsequently map the result back into our batch, we will use the [`archive`](https://docs.redpanda.com/connect/components/processors/archive/) and [`unarchive`](https://docs.redpanda.com/connect/components/processors/unarchive/) processors in our [`branch`](https://docs.redpanda.com/connect/components/processors/branch/) flow, like this: ```yaml pipeline: processors: - branch: request_map: 'root.text = this.article.content' processors: - archive: format: json_array - http: url: http://localhost:4198/hyperbole verb: POST - unarchive: format: json_array result_map: 'root.tmp.hyperbole_rank = this.hyperbole_rank' ``` The purpose of the `json_array` format `archive` processor is to take a batch of JSON documents and place them into a single document as an array. Subsequently, we then send one single request for each batch. After the request is made we do the opposite with the `unarchive` processor in order to convert it back into a batch of the original size. ### [](#fake-news-detector)Fake news detector Finally, we are going to use a 'fake news detector' that takes the article contents as well as the output of the previous two enrichments and calculates a fake news rank between 0 and 1. This service behaves similarly to the claims detector service and takes a document of the form: ```json { "text": "The world was shocked this morning to find that all dogs have stopped barking.", "hyperbole_rank": 0.73, "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } ``` And returns an object of the form: ```json { "fake_news_rank": 0.893 } ``` We then wish to map the field `fake_news_rank` from that result into the original document at the path `article.fake_news_score`. Our [`branch`](https://docs.redpanda.com/connect/components/processors/branch/) block for this enrichment would look like this: ```yaml pipeline: processors: - branch: request_map: | root.text = this.article.content root.claims = this.tmp.claims root.hyperbole_rank = this.tmp.hyperbole_rank processors: - http: url: http://localhost:4199/fakenews verb: POST result_map: 'root.article.fake_news_score = this.fake_news_rank' ``` Note that in our `request_map` we are targeting fields that are populated from the previous two enrichments. If we were to execute all three enrichments in a sequence we’ll end up with a document looking like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking.", "fake_news_score": 0.76 }, "tmp": { "hyperbole_rank": 0.34, "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } } ``` Great! However, as a streaming pipeline this set up isn’t ideal as our first two enrichments are independent and could potentially be executed in parallel in order to reduce processing latency. ## [](#combining-into-a-workflow)Combining into a workflow If we configure our enrichments within a [`workflow` processor](https://docs.redpanda.com/connect/components/processors/workflow/) we can use Redpanda Connect to automatically detect our dependency graph, giving us two key benefits: 1. Enrichments at the same level of a dependency graph (claims and hyperbole) will be executed in parallel. 2. When introducing more enrichments to our pipeline the added complexity of resolving the dependency graph is handled automatically by Redpanda Connect. Placing our branches within a [`workflow` processor](https://docs.redpanda.com/connect/components/processors/workflow/) makes our final pipeline configuration look like this: ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group batching: count: 20 # Tune this to set the size of our document batches. period: 1s pipeline: processors: - workflow: meta_path: '' # Don't bother storing branch metadata. branches: claims: request_map: 'root.text = this.article.content' processors: - http: url: http://localhost:4197/claims verb: POST result_map: 'root.tmp.claims = this.claims' hyperbole: request_map: 'root.text = this.article.content' processors: - archive: format: json_array - http: url: http://localhost:4198/hyperbole verb: POST - unarchive: format: json_array result_map: 'root.tmp.hyperbole_rank = this.hyperbole_rank' fake_news: request_map: | root.text = this.article.content root.claims = this.tmp.claims root.hyperbole_rank = this.tmp.hyperbole_rank processors: - http: url: http://localhost:4199/fakenews verb: POST result_map: 'root.article.fake_news_score = this.fake_news_rank' - catch: - log: fields_mapping: 'root.content = content().string()' message: "Enrichments failed due to: ${!error()}" - mapping: | root = this root.tmp = deleted() output: kafka: addresses: [ TODO ] topic: comments_hydrated ``` Since the contents of `tmp` won’t be required downstream we remove it after our enrichments using a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/). A [`catch`](https://docs.redpanda.com/connect/components/processors/catch/) processor was added at the end of the pipeline which catches documents that failed enrichment. You can replace the log event with a wide range of recovery actions such as sending to a dead-letter/retry queue, dropping the message entirely, etc. You can read more about error handling [in this article](https://docs.redpanda.com/connect/configuration/error_handling/). --- # Page 352: Filtering and Sampling **URL**: https://docs.redpanda.com/connect/cookbooks/filtering.md --- # Filtering and Sampling --- title: Filtering and Sampling latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: filtering page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: filtering.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/filtering.adoc description: Configure Redpanda Connect to conditionally drop messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Filtering events in Redpanda Connect is both easy and flexible, this cookbook demonstrates a few different types of filtering you can do. All of these examples make use of the [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) but shouldn’t require any prior knowledge. ## [](#the-basic-filter)The basic filter Dropping events with [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) is done by mapping the function `deleted()` to the `root` of the mapped document. To remove all events indiscriminately you can simply do: ```yaml pipeline: processors: - mapping: root = deleted() ``` But that’s most likely not what you want. We can instead only delete an event under certain conditions with a [`match`](https://docs.redpanda.com/connect/guides/bloblang/about/#pattern-matching) or [`if`](https://docs.redpanda.com/connect/guides/bloblang/about/#conditional-mapping) expression: ```yaml pipeline: processors: - mapping: | root = if @topic.or("") == "foo" || this.doc.type == "bar" || this.doc.urls.contains("https://www.benthos.dev/").catch(false) { deleted() } ``` The above config removes any events where: - The metadata field `topic` is equal to `foo` - The event field `doc.type` (a string) is equal to `bar` - The event field `doc.urls` (an array) contains the string `https://www.benthos.dev/` Events that do not match any of these conditions will remain unchanged. ## [](#sample-events)Sample events Another type of filter we might want is a sampling filter, we can do that with a random number generator: ```yaml pipeline: processors: - mapping: | # Drop 50% of documents randomly root = if random_int() % 2 == 0 { deleted() } ``` We can also do this in a deterministic way by hashing events and filtering by that hash value: ```yaml pipeline: processors: - mapping: | # Drop ~10% of documents deterministically (same docs filtered each run) root = if content().hash("xxhash64").slice(-8).number() % 10 == 0 { deleted() } ``` --- # Page 353: Stream data to BigQuery with the Storage Write API **URL**: https://docs.redpanda.com/connect/cookbooks/gcp_bigquery_write_api.md --- # Stream data to BigQuery with the Storage Write API --- title: Stream data to BigQuery with the Storage Write API latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: gcp_bigquery_write_api page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: gcp_bigquery_write_api.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/gcp_bigquery_write_api.adoc description: Configure Redpanda Connect to stream, upsert, and delete data in BigQuery using the Storage Write API. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Find patterns for streaming JSON into BigQuery learning-objective-2: Look up CDC upsert and delete patterns for BigQuery learning-objective-3: Identify auto-create, multi-table routing, and throughput-tuning patterns page-git-created-date: "2026-06-16" page-git-modified-date: "2026-09-04" --- The [`gcp_bigquery_write_api` output](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/) streams messages into BigQuery using the [Storage Write API](https://cloud.google.com/bigquery/docs/write-api), which offers higher throughput and lower latency than the legacy streaming API or load jobs. This cookbook provides reusable patterns for streaming inserts, change data capture (CDC) upserts and deletes, automatic table creation, multi-table routing, and throughput tuning. Use this cookbook to: - Find patterns for streaming JSON into BigQuery - Look up CDC upsert and delete patterns for BigQuery - Identify auto-create, multi-table routing, and throughput-tuning patterns ## [](#prerequisites)Prerequisites Before using these patterns, ensure you have the following configured: ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#enterprise-license)Enterprise license The `gcp_bigquery_write_api` output is an enterprise component. To use it, you need a Redpanda Connect Enterprise license. See [Enterprise Licensing](https://docs.redpanda.com/connect/get-started/licensing/). ### [](#bigquery-dataset-and-permissions)BigQuery dataset and permissions You need a Google Cloud project with a BigQuery dataset, and a service account with permission to write to (and, for the auto-create pattern, create) tables in that dataset. The BigQuery Data Editor (`roles/bigquery.dataEditor`) role covers both. The output loads credentials from the standard [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) chain. You can also supply a service account key directly with the [`credentials_json`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#credentials_json) field. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables to keep configuration separate from credentials. ```bash export GCP_PROJECT=my-project (1) export BQ_DATASET=analytics (2) export BQ_TABLE=events (3) export REDPANDA_BROKERS=localhost:9092 (4) export SOURCE_TOPIC=events (5) ``` | 1 | The Google Cloud project ID that owns the dataset. | | --- | --- | | 2 | The BigQuery dataset that holds the destination tables. | | 3 | The destination table for single-table examples. | | 4 | The Redpanda broker addresses for the source topic. | | 5 | The Redpanda topic to consume from. | ## [](#stream-json-to-bigquery)Stream JSON to BigQuery The simplest pattern consumes JSON messages from a Redpanda topic and streams them into a single table using the default stream, which provides at-least-once delivery with the lowest latency: ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # BigQuery's proto3 JSON mapping encodes int64/uint64 as strings, so # convert integer columns to strings before writing. - mapping: | root = this root.id = this.id.string() output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: ${BQ_TABLE} message_format: json write_mode: default_stream batching: count: 1000 period: 5s ``` > ⚠️ **WARNING** > > The proto3 JSON mapping encodes `int64` and `uint64` values as strings. When a message maps to an `INTEGER` (`INT64`) column, the JSON field must use a string value, for example `"age": "30"` rather than `"age": 30`. Otherwise the write fails with an unmarshalling error. For the full list of message formats and connection options, see the [connector reference](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/). ## [](#guarantee-exactly-once-delivery-per-batch)Guarantee exactly-once delivery per batch To commit each batch atomically with exactly-once semantics, set [`write_mode`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#write_mode) to `pending_stream`. Each batch is written to a per-batch pending stream that commits as a single unit: ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # Project only the columns the table expects and convert the integer id to # a string for BigQuery's proto3 JSON mapping. - mapping: | root.id = this.id.string() root.event = this.event output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: ${BQ_TABLE} message_format: json write_mode: pending_stream batching: count: 5000 period: 10s ``` Exactly-once applies within a single committed batch. Tune batch size to balance latency against commit overhead. See [Message Batching](https://docs.redpanda.com/connect/configuration/batching/). ## [](#create-the-destination-table-automatically)Create the destination table automatically Set [`auto_create_table`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#auto_create_table) to `true` to have Redpanda Connect create the table if it does not exist, using the supplied [`schema`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#schema). You can also define partitioning and clustering for the new table: ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # Shape each message to match the table schema. The id column is INTEGER, # so send it as a string for BigQuery's proto3 JSON mapping. - mapping: | root.id = this.id.string() root.name = this.name root.created_at = this.created_at output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: ${BQ_TABLE} message_format: json auto_create_table: true schema: - name: id type: INTEGER mode: REQUIRED - name: name type: STRING - name: created_at type: TIMESTAMP mode: REQUIRED time_partitioning: type: DAY field: created_at clustering: - id batching: count: 1000 period: 5s ``` Each schema column accepts a `name`, `type`, and optional `mode` (`NULLABLE`, `REQUIRED`, or `REPEATED`). Partitioning and clustering apply only when the table is created. ## [](#apply-cdc-upserts)Apply CDC upserts To keep a BigQuery table in sync with a changing source, use change data capture (CDC) upserts. Set [`write_mode`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#write_mode) to `upsert`, declare the [`primary_keys`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#primary_keys), and provide a [`change_type`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#change_type) expression. BigQuery injects the `_CHANGE_TYPE` pseudo-column for each row: ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # Stringify the integer primary key for BigQuery's proto3 JSON mapping. - mapping: | root = this root.id = this.id.string() output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: ${BQ_TABLE} message_format: json write_mode: upsert primary_keys: - id change_type: '${! "UPSERT" }' auto_create_table: true schema: - name: id type: INTEGER mode: REQUIRED - name: name type: STRING - name: updated_at type: TIMESTAMP batching: count: 1000 period: 5s ``` In `upsert` mode, `change_type` must resolve to `UPSERT` for every row. The target table must have a primary key: when `auto_create_table` is `true`, `primary_keys` defines it; for a pre-existing table, the output falls back to the `PRIMARY KEY` declared on the table. Up to 16 columns are supported, and composite keys are matched in the order listed. > 📝 **NOTE** > > CDC modes write through the default stream, as required by BigQuery, so they provide at-least-once delivery rather than the per-batch exactly-once semantics of `pending_stream`. ## [](#handle-inserts-updates-and-deletes)Handle inserts, updates, and deletes To propagate deletes as well as upserts, set [`write_mode`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#write_mode) to `upsert_delete` and resolve `change_type` to either `UPSERT` or `DELETE` per row (case-insensitive). The following example derives the change type from a source operation field in a `mapping` processor, stores it in metadata, and references it from the output. It also uses [`change_sequence_number`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#change_sequence_number) to control ordering when rows for the same key can arrive out of order: ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # Derive the CDC change type and sequence number from the source event. # This example assumes a Debezium-style envelope with an "op" field # ("c"/"u" for upsert, "d" for delete), the row in "after", and a log # sequence number in "lsn". - mapping: | root = this.after root.id = this.after.id.string() meta change_type = if this.op == "d" { "DELETE" } else { "UPSERT" } meta change_seq = this.lsn.string() output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: ${BQ_TABLE} message_format: json write_mode: upsert_delete primary_keys: - id change_type: '${! metadata("change_type") }' # Order changes by a monotonic sequence number so a late-arriving older # change cannot overwrite a newer row. change_sequence_number: '${! metadata("change_seq") }' batching: count: 1000 period: 5s ``` When `change_sequence_number` is unset, BigQuery resolves ordering by arrival time. Set it from a monotonic source value (such as a log sequence number) so that a late-arriving older change cannot overwrite a newer one. ## [](#route-to-multiple-tables)Route to multiple tables The [`table`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#table) field supports interpolation, so you can route messages to different tables based on their content or metadata. The table name is resolved from the first message in each batch, so batch by the routing key to keep each batch single-table: ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # Stringify the integer id for BigQuery's proto3 JSON mapping. - mapping: | root = this root.id = this.id.string() # Split each incoming batch into single-table batches keyed by event_type. # The output resolves the table from the first message in each batch, so # every batch must contain rows for one table only. - group_by_value: value: '${! json("event_type") }' output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: '${! json("event_type") }' message_format: json # No batching block here: this preserves the single-table batches produced # by group_by_value. Adding an output batching policy would re-mix tables. max_in_flight: 8 max_cached_streams: 1024 ``` When routing across many tables, raise [`max_cached_streams`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#max_cached_streams) so the output can keep a write stream open per active table. ## [](#tune-batching-and-throughput)Tune batching and throughput Throughput depends on batch size and the number of concurrent in-flight requests. Increase [`max_in_flight`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#max_in_flight) to send more batches in parallel, and size batches with a [batching policy](https://docs.redpanda.com/connect/configuration/batching/): ```yaml input: kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["${SOURCE_TOPIC}"] consumer_group: bigquery_sink pipeline: processors: # Stringify the integer id for BigQuery's proto3 JSON mapping. - mapping: | root = this root.id = this.id.string() output: gcp_bigquery_write_api: project: ${GCP_PROJECT} dataset: ${BQ_DATASET} table: ${BQ_TABLE} message_format: json write_mode: default_stream # Send more batches concurrently for higher throughput. max_in_flight: 16 # Larger batches reduce per-request overhead at the cost of latency. batching: count: 10000 byte_size: 5000000 period: 10s ``` ## [](#troubleshoot)Troubleshoot ### [](#writes-fail-with-an-unmarshalling-error)Writes fail with an unmarshalling error BigQuery rejects integer fields supplied as JSON numbers. Ensure `int64` and `uint64` values are encoded as strings, for example `"count": "42"`. Use a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) to convert numeric fields to strings before the output if your source emits them as numbers. ### [](#schema-resolution-times-out)Schema resolution times out The output fetches table metadata before writing. If you route to many tables or a backend is slow, increase [`schema_resolve_timeout`](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/#schema_resolve_timeout) (default `15s`). On the `auto_create_table` path this budget also covers table creation, so allow extra headroom. ### [](#cdc-writes-are-rejected)CDC writes are rejected For `upsert` and `upsert_delete` modes, confirm the target table has a primary key and that `change_type` resolves to a valid value (`UPSERT`, or `DELETE` for `upsert_delete`). Rows that resolve to an invalid change type are rejected by BigQuery. ## [](#suggested-reading)Suggested reading - [`gcp_bigquery_write_api` output reference](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api/) - [Batching](https://docs.redpanda.com/connect/configuration/batching/) - [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) - [BigQuery change data capture](https://cloud.google.com/bigquery/docs/change-data-capture) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 354: Google Cloud Spanner CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/gcp_spanner_cdc.md --- # Google Cloud Spanner CDC Patterns --- title: Google Cloud Spanner CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: gcp_spanner_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: gcp_spanner_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/gcp_spanner_cdc.adoc description: Learn how to capture, filter, transform, and route Google Cloud Spanner change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing Google Cloud Spanner CDC events learning-objective-2: Adapt integration patterns to route CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming Spanner change records page-git-created-date: "2026-06-17" page-git-modified-date: "2026-06-17" --- The `gcp_spanner_cdc` input captures row-level changes from Google Cloud Spanner tables using [Spanner change streams](https://cloud.google.com/spanner/docs/change-streams). Use these patterns to filter, transform, and route Spanner change records to Redpanda, Amazon S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing Google Cloud Spanner CDC events - Adapt integration patterns to route CDC data to Redpanda and S3 - Identify patterns for filtering and transforming Spanner change records > 📝 **NOTE** > > The `gcp_spanner_cdc` input is in beta. The API is subject to change. ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [rpk installation](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#spanner-change-stream)Spanner change stream The `gcp_spanner_cdc` input reads from a Spanner change stream. The change stream must be created in your Spanner database before running the pipeline. To create a change stream that tracks all tables: ```sql CREATE CHANGE STREAM AllChanges FOR ALL; ``` To track specific tables: ```sql CREATE CHANGE STREAM OrderChanges FOR orders, customers; ``` For more information, see the [Spanner change streams documentation](https://cloud.google.com/spanner/docs/change-streams/manage). ### [](#google-cloud-platform-gcp-authentication)Google Cloud Platform (GCP) authentication The `gcp_spanner_cdc` input supports two authentication methods: - **Application Default Credentials (ADC)**: The recommended approach for GCP-native environments (Cloud Run, GKE, Compute Engine). No configuration needed when running with a service account attached to the instance. - **Service account key**: Set the `credentials_json` field to a base64-encoded service account key JSON file. Use this approach when running outside GCP or when ADC is not available: ```bash export GCP_CREDENTIALS="$(base64 < /path/to/service-account.json | tr -d '\n')" ``` Then reference it in your pipeline config: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} credentials_json: ${GCP_CREDENTIALS} ``` ### [](#progress-tracking)Progress tracking The `gcp_spanner_cdc` input stores progress in a Spanner metadata table (`cdc_metadata_` by default). No external cache is required. The pipeline resumes from the last acknowledged change record after a restart. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export GCP_PROJECT_ID="analytics-prod" (1) export GCP_INSTANCE_ID="primary" (2) export GCP_DATABASE_ID="orders" (3) export GCP_STREAM_ID="AllChanges" (4) export REDPANDA_BROKERS=localhost:9092 (5) export S3_BUCKET=cdc-archive (6) ``` | 1 | The GCP project ID. | | --- | --- | | 2 | The Spanner instance ID. | | 3 | The Spanner database ID. | | 4 | The name of the change stream to read from. | | 5 | The Redpanda broker addresses (for Redpanda output examples). | | 6 | The Amazon S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change records from a Spanner change stream and outputs them with key fields: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} pipeline: processors: - mapping: | root.mod_type = meta("mod_type") root.table_name = meta("table_name") root.commit_timestamp = meta("commit_timestamp") root.data = this root.transaction_id = meta("server_transaction_id") output: stdout: codec: lines ``` For details on the change record structure and available fields, see the [Spanner change record reference](https://cloud.google.com/spanner/docs/change-streams/details#record-types). The `gcp_spanner_cdc` input emits one message per data change. Fields such as `mod_type` (INSERT, UPDATE, or DELETE), `table_name`, `commit_timestamp`, and `server_transaction_id` are available as message metadata (use `meta("field_name")`). The message body contains the row data with `keys`, `new_values`, and `old_values`. ## [](#filter-cdc-events)Filter CDC events The `gcp_spanner_cdc` input supports built-in filtering with the `allowed_mod_types` field. This filters events at the source before they reach your pipeline, which is more efficient than post-ingestion filtering: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} allowed_mod_types: - INSERT - UPDATE pipeline: processors: - mapping: | root.mod_type = meta("mod_type") root.table_name = meta("table_name") root.commit_timestamp = meta("commit_timestamp") root.new_values = this.new_values output: stdout: codec: lines ``` This pattern: - Captures only `INSERT` and `UPDATE` changes at the input level, ignoring `DELETE` events entirely - Extracts new values from each change record ## [](#route-to-redpanda)Route to Redpanda Stream Spanner changes to Redpanda, using the table name as the topic: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} pipeline: processors: - mapping: | meta topic = meta("table_name") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } key: ${! meta("server_transaction_id") } batching: count: 100 period: 1s ``` This pattern: - Uses the table name as the Redpanda topic - Batches messages for efficient delivery - Sets the message key to `server_transaction_id`, grouping related changes together ## [](#route-to-s3)Route to S3 Archive change records to Amazon S3 for long-term storage and analytics: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} pipeline: processors: - mapping: | root.mod_type = meta("mod_type") root.table_name = meta("table_name") root.commit_timestamp = meta("commit_timestamp") root.data = this root.transaction_id = meta("server_transaction_id") output: aws_s3: bucket: ${S3_BUCKET} path: >- cdc/${! meta("table_name") }/${! timestamp_unix().format_timestamp("2006/01/02/15") }/${! uuid_v4() }.ndjson batching: count: 1000 period: 5m processors: - archive: format: lines ``` This pattern: - Organizes files by table name and time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different change types to different Redpanda topics: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} output: switch: cases: - check: meta("mod_type") == "INSERT" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: spanner.inserts - output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: spanner.changes ``` This pattern: - Routes `INSERT` events to one Redpanda topic and all other changes (`UPDATE`, `DELETE`) to another - Supports specialized downstream consumers per operation type ## [](#stream-a-time-range)Stream a time range Use `start_timestamp` and `end_timestamp` to process a specific window of changes. Both fields accept RFC 3339 timestamps: ```yaml input: gcp_spanner_cdc: project_id: ${GCP_PROJECT_ID} instance_id: ${GCP_INSTANCE_ID} database_id: ${GCP_DATABASE_ID} stream_id: ${GCP_STREAM_ID} start_timestamp: "2024-01-01T00:00:00Z" (1) end_timestamp: "2024-01-02T00:00:00Z" (2) ``` | 1 | Start reading from this timestamp. Defaults to the current time if not set. | | --- | --- | | 2 | Stop reading at this timestamp. Omit to stream continuously. | ## [](#troubleshoot-common-issues)Troubleshoot common issues Use these steps to diagnose and fix common problems with the `gcp_spanner_cdc` input. ### [](#no-events-received)No events received If no events arrive: 1. Verify the change stream exists in the database: ```sql SELECT * FROM INFORMATION_SCHEMA.CHANGE_STREAMS WHERE CHANGE_STREAM_NAME = 'AllChanges'; ``` 2. Confirm the service account has the required IAM roles: - `roles/spanner.databaseReader` on the Spanner database - `roles/spanner.databaseUser` to create and read the metadata table 3. Check that `stream_id` matches the exact name of the change stream. ### [](#metadata-table-creation-fails)Metadata table creation fails If the pipeline fails because it cannot create the metadata table: - Confirm the service account has `roles/spanner.databaseUser` on the database. - Use the `metadata_table` field to specify an existing table if you prefer not to create a new one. ### [](#duplicate-events-after-restart)Duplicate events after restart The `gcp_spanner_cdc` input provides at-least-once delivery. If the pipeline fails mid-stream, it may replay some records on restart. To handle duplicates: - Use idempotent processing in downstream systems. - Deduplicate using the `server_transaction_id` and `record_sequence` fields from the change record. ## [](#next-steps)Next steps - [Google Cloud Spanner CDC input reference](https://docs.redpanda.com/connect/components/inputs/gcp_spanner_cdc/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 355: Work with Jira Issues **URL**: https://docs.redpanda.com/connect/cookbooks/jira.md --- # Work with Jira Issues --- title: Work with Jira Issues latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: jira page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: jira.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/jira.adoc description: Learn how to use the Jira processor as both an input and output in Redpanda Connect. page-git-created-date: "2025-11-21" page-git-modified-date: "2026-01-05" --- The Jira processor enables querying Jira issues using JQL (Jira Query Language) and returning structured data. Unlike traditional connectors, Jira is implemented as a **processor only**, which can be used as both an input (with `generate`) and an output (with `drop`). ## [](#prerequisites)Prerequisites ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#enterprise-license)Enterprise license The Jira processor requires a Redpanda Enterprise Edition license. See [Enterprise Licensing](https://docs.redpanda.com/connect/get-started/licensing/) for instructions on generating a trial license and setting the `REDPANDA_LICENSE` environment variable. ### [](#jira-environment-variables)Jira environment variables The examples in this cookbook use environment variables for Jira credentials and configuration. This allows you to keep sensitive credentials secure and separate from your pipeline configuration files. 1. [Generate a Jira API token](https://id.atlassian.com/manage-profile/security/api-tokens). 2. Set up your environment variables: ```bash JIRA_BASE_URL=https://.atlassian.net (1) JIRA_USERNAME= (2) JIRA_API_TOKEN= (3) JIRA_PROJECT= (4) ``` | 1 | Your Jira instance URL. You can find this in your browser when logged into Jira. | | --- | --- | | 2 | Your Jira account email address. | | 3 | The API token generated from your Atlassian account. | | 4 | The key of the Jira project you want to query (such as DOC or K8S). | You can export these variables in your shell or place them in a `.env` file for use with `rpk connect run --env-file `. ## [](#use-jira-as-an-input)Use Jira as an input To use Jira as an input, combine the `generate` input with the Jira processor. This pattern triggers Jira queries at regular intervals or on-demand. ### [](#query-jira-periodically)Query Jira periodically This example queries Jira every 30 seconds for recent issues: ```yaml input: generate: mapping: | root.jql = "project = ${JIRA_PROJECT} AND created >= -1d ORDER BY created DESC" root.maxResults = 50 root.fields = ["key", "summary", "status", "assignee", "created"] interval: 30s pipeline: processors: - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 50 timeout: 30s backoff: max_retries: 10 # Transform each issue - mapping: | root.issue_key = this.key root.summary = this.fields.summary root.status = this.fields.status.name root.assignee = if this.fields.assignee != null { this.fields.assignee.displayName } else { "Unassigned" } root.created = this.fields.created output: stdout: codec: lines ``` ### [](#one-time-query)One-time query For a single query, use `count` instead of `interval`: ```yaml input: generate: mapping: | root.jql = "project = ${JIRA_PROJECT} ORDER BY created DESC" root.maxResults = 100 root.fields = ["key", "summary", "status"] count: 1 # Run only once pipeline: processors: - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" output: stdout: {} ``` ### [](#advanced-query-with-custom-fields)Advanced query with custom fields ```yaml input: generate: mapping: | root.jql = "assignee = currentUser() AND status IN (Open, 'In Progress') ORDER BY updated DESC" root.maxResults = 20 root.fields = [ "key", "summary", "status", "assignee", "priority", "created", "updated", "customfield_10001" # Custom field ID ] root.startAt = 0 interval: 60s pipeline: processors: - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 100 # Transform each individual issue message - mapping: | root.key = this.key root.summary = this.fields.summary root.status = this.fields.status.name root.priority = this.fields.priority.name root.custom_field = this.fields.customfield_10001 output: kafka: addresses: ["localhost:9092"] topic: jira-issues ``` ## [](#use-jira-as-an-output)Use Jira as an output To use Jira as an output, place the Jira processor before a `drop` output. This executes queries without forwarding results downstream. ### [](#query-and-drop-results)Query and drop results ```yaml input: stdin: codec: lines pipeline: processors: # Prepare Jira query from incoming message - mapping: | root.jql = this.jql.or("project = ${JIRA_PROJECT} AND status = Open") root.maxResults = this.maxResults.or(100) root.fields = this.fields.or(["key", "summary", "status"]) # Execute Jira query - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" backoff: max_retries: 3 timeout: 30s # Log each issue before dropping - log: message: "Processing Jira issue: ${! this.key } - ${! this.fields.summary }" level: INFO output: drop: {} ``` ### [](#trigger-queries-from-kafka)Trigger queries from Kafka ```yaml input: kafka: addresses: ["localhost:9092"] topics: ["jira-query-requests"] consumer_group: jira-query-consumer pipeline: processors: # Parse incoming query request - mapping: | root.jql = this.query root.maxResults = 50 # Execute query - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" # Process each individual issue - mapping: | root.query_id = uuid_v4() root.timestamp = now() root.issue_key = this.key root.issue_summary = this.fields.summary output: drop: {} ``` ## [](#input-message-format)Input message format The Jira processor expects input messages containing valid Jira queries in JSON format: ```json { "jql": "project = MYPROJECT AND status = Open", "maxResults": 50, "fields": ["key", "summary", "status", "assignee"] } ``` ### [](#required-fields)Required fields - `jql`: The JQL (Jira Query Language) query string ### [](#optional-fields)Optional fields - `maxResults`: Maximum number of results to return (default: 50) - `fields`: Array of field names to include in the response ## [](#configuration-options)Configuration options For a complete list of configuration options, see [Jira Processor Reference](https://docs.redpanda.com/connect/components/processors/jira/). ## [](#jql-query-language)JQL query language The Jira processor supports querying Jira issues using JQL (Jira Query Language). JQL is a flexible, text-based query language similar to SQL that allows you to search for issues matching specific criteria. ### [](#common-jql-patterns)Common JQL patterns Here are common JQL patterns for filtering issues: #### [](#recent-issues-by-project)Recent issues by project ```jql project = AND created >= -7d ORDER BY created DESC ``` #### [](#issues-assigned-to-current-user)Issues assigned to current user ```jql assignee = currentUser() AND status != Done ``` #### [](#issues-by-status)Issues by status ```jql project = AND status IN (Open, 'In Progress', 'To Do') ``` #### [](#issues-by-priority)Issues by priority ```jql project = AND priority = High ORDER BY created DESC ``` #### [](#issues-updated-recently)Issues updated recently ```jql project = AND updated >= -1d ORDER BY updated DESC ``` ## [](#run-the-examples)Run the examples Each example in this cookbook can be saved as a YAML file and run individually. The examples use environment variables for credentials and configuration, so you don’t need to modify the YAML files. 1. Set up your environment: ```bash # Set Jira credentials export JIRA_BASE_URL=https://your-domain.atlassian.net export JIRA_USERNAME=your-email@example.com export JIRA_API_TOKEN=your-api-token export JIRA_PROJECT=YOUR_PROJECT ``` 2. Save any example to a YAML file (for example, `jira-query.yaml`). 3. Run the example: ```bash rpk connect run jira-query.yaml ``` ## [](#output-message-format)Output message format The Jira processor returns **individual issue messages**, rather than a response object with an `issues` array. Each message output by the Jira processor represents a single issue: ```json { "id": "12345", "key": "DOC-123", "fields": { "summary": "Example issue", "status": { "name": "In Progress" }, "assignee": { "displayName": "John Doe" }, "created": "2025-01-15T10:30:00.000-0800", "updated": "2025-01-16T14:20:00.000-0800" } } ``` ## [](#process-output-messages)Process output messages When processing messages from the Jira processor, keep in mind: - **Do not** use `unarchive` to split an `issues` array - **Do not** reference `this.issues` or `this.total` in your mappings - **Do** access fields directly such as `this.key`, `this.fields.summary`. ### [](#handle-null-fields)Handle null fields Some Jira fields can be null and require conditional handling: ```yaml pipeline: processors: - mapping: | # Assignee can be null root.assignee = if this.fields.assignee != null { this.fields.assignee.displayName } else { "Unassigned" } # Reporter can be null root.reporter = if this.fields.reporter != null { this.fields.reporter.displayName } else { "Unknown" } # Resolution date is null for open issues root.resolved_at = this.fields.resolutiondate # Calculate lead time only for resolved issues root.lead_time_days = if this.fields.resolutiondate != null { (this.fields.resolutiondate.ts_parse("2006-01-02T15:04:05.999-0700").ts_unix() - this.fields.created.ts_parse("2006-01-02T15:04:05.999-0700").ts_unix()) / 86400 } else { null } ``` The Jira processor automatically handles pagination internally using the `max_results_per_page` setting. The processor: 1. Makes the initial request with `startAt=0`. 2. Checks if more results are available. 3. Automatically fetches subsequent pages until all results are retrieved. 4. Outputs each issue as an individual message. You don’t need to handle pagination manually. The processor streams all matching issues as separate messages. ## [](#error-handling)Error handling The Jira processor includes built-in retry logic for rate limiting (HTTP 429) and configurable timeouts. See [Jira Processor Reference](https://docs.redpanda.com/connect/components/processors/jira/) for detailed configuration options. ## [](#create-and-update-jira-issues)Create and update Jira issues The Jira processor is **read-only** and only supports querying. To create or update Jira issues, you must use the `http` processor with the Jira REST API. ### [](#create-a-jira-issue)Create a Jira issue Use the HTTP processor to POST to the Jira REST API `/issue` endpoint. This example creates a Task issue with a title and description: ```yaml # Create Jira Issue with HTTP Processor # # The Jira processor is read-only and cannot create or update issues. # Use the HTTP processor to create Jira issues using the REST API. input: generate: mapping: | root = { "title": "Example issue from Redpanda Connect", "description": "This issue was created using the HTTP processor", "issue_type": "Task" } count: 1 pipeline: processors: - mapping: | root = { "fields": { "project": { "key": "${JIRA_PROJECT}" }, "summary": this.title, "description": { "type": "doc", "version": 1, "content": [ { "type": "paragraph", "content": [ { "type": "text", "text": this.description } ] } ] }, "issuetype": { "name": this.issue_type } } } - label: create_issue http: url: "${JIRA_BASE_URL}/rest/api/3/issue" verb: POST headers: Content-Type: application/json Accept: application/json basic_auth: enabled: true username: "${JIRA_USERNAME}" password: "${JIRA_API_TOKEN}" - mapping: | root.issue_key = this.key root.issue_id = this.id root.self_link = this.self output: stdout: codec: lines tests: - name: Transform data for creating issue target_processors: '/pipeline/processors/0' environment: JIRA_PROJECT: "TEST" input_batch: - json_content: title: "Test issue" description: "Test description" issue_type: "Bug" output_batches: - - json_equals: fields: project: key: "TEST" summary: "Test issue" description: type: "doc" version: 1 content: - type: "paragraph" content: - type: "text" text: "Test description" issuetype: name: "Bug" - name: Mock create issue API response target_processors: '/pipeline/processors' mocks: create_issue: mapping: | root.id = "10001" root.key = "DOC-789" root.self = "https://your-domain.atlassian.net/rest/api/3/issue/10001" input_batch: - json_content: title: "Test issue" description: "Test description" issue_type: "Task" output_batches: - - json_equals: issue_key: "DOC-789" issue_id: "10001" self_link: "https://your-domain.atlassian.net/rest/api/3/issue/10001" ``` - **Atlassian Document Format (ADF)**: The `description` field uses ADF, Jira’s rich text format. For simple text, wrap it in a paragraph structure as shown. - **Issue type**: Common types are `Task`, `Bug`, `Story`, `Epic`. The type must exist in your Jira project. - **Required fields**: At minimum, you need `project`, `summary`, and `issuetype`. Other fields may be required based on your Jira configuration. - **Response**: The API returns the created issue’s `key`, `id`, and `self` (URL). ## [](#troubleshoot)Troubleshoot ### [](#license-errors)License errors If you receive license-related errors: 1. Verify the license is exported: ```bash echo $REDPANDA_LICENSE ``` 2. Regenerate if needed: ```bash rpk generate license --name "" \ --last-name "" \ --email "" \ --company "" export REDPANDA_LICENSE=$(cat ./redpanda.license) ``` 3. Check license expiration: The trial license expires after 30 days. Check the output from `rpk generate license` for the expiration date. ### [](#authentication-errors)Authentication errors If you receive 401 Unauthorized errors: - Verify your API token is correct - Ensure you’re using your email address as the username - Check that your Jira instance URL is correct (include `https://`) ### [](#rate-limiting)Rate limiting If you hit Jira rate limits (HTTP 429): - Increase `max_retries` in the Jira processor config - Reduce query frequency by increasing `interval` - Use more specific JQL queries to reduce result sizes ### [](#query-performance)Query performance For large result sets: - Use more specific JQL queries - Limit fields with the `fields` parameter - Reduce `max_results_per_page` if memory is constrained ## [](#see-also)See also - [Jira Processor Reference](https://docs.redpanda.com/connect/components/processors/jira/) - [Jira REST API Documentation](https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/) - [JQL Query Guide](https://www.atlassian.com/software/jira/guides/jql) - [Bloblang Interpolation](https://docs.redpanda.com/connect/configuration/interpolation/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 356: Joining Streams **URL**: https://docs.redpanda.com/connect/cookbooks/joining_streams.md --- # Joining Streams --- title: Joining Streams latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: joining_streams page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: joining_streams.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/joining_streams.adoc description: How to hydrate documents by joining multiple streams. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- This cookbook demonstrates how to merge JSON events from parallel streams using content based rules and a [cache](https://docs.redpanda.com/connect/components/caches/about/) of your choice. The imaginary problem we are going to solve is hydrating a feed of article comments with information from their parent articles. We will be consuming and writing to Kafka, but the example works with any [input](https://docs.redpanda.com/connect/components/inputs/about/) and [output](https://docs.redpanda.com/connect/components/outputs/about/) combination. Articles are received over the topic `articles` and look like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Good article", "content": "this is a totally good article" }, "user": { "id": "user1" } } ``` Comments can either be posted on an article or a parent comment, are received over the topic `comments`, and look like this: ```json { "type": "comment", "comment": { "id": "456bar", "parent_id": "123foo", "content": "this article is bad" }, "user": { "id": "user2" } } ``` Our goal is to end up with a single stream of comments, where information about the root article of the comment is attached to the event. The above comment should exit our pipeline looking like this: ```json { "type": "comment", "comment": { "id": "456bar", "parent_id": "123foo", "content": "this article is bad" }, "article": { "title": "Good article", "content": "this is a totally good article" }, "user": { "id": "user2" } } ``` In order to achieve this we will need to cache articles as they pass through our pipelines and then retrieve them for each comment passing through. Since the parent of a comment might be another comment we will also need to cache and retrieve comments in the same way. ## [](#caching-articles)Caching articles Our first pipeline is very simple, we just consume articles, reduce them to only the fields we wish to cache, and then cache them. If we receive the same article multiple times we’re going to assume it’s okay to overwrite the old article in the cache. In this example I’m targeting Redis, but you can choose any of the supported [cache targets](https://docs.redpanda.com/connect/components/caches/about/). The TTL of cached articles is set to one week. ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group pipeline: processors: # Reduce document into only fields we wish to cache. - mapping: 'article = article' # Store reduced articles into our cache. - cache: operator: set resource: hydration_cache key: '${!json("article.id")}' value: '${!content()}' # Drop all articles after they are cached. output: drop: {} cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` ## [](#hydrating-comments)Hydrating comments Our second pipeline consumes comments, caches them in case a subsequent comment references them, obtains its parent (article or comment), and attaches the root article to the event before sending it to our output topic `comments_hydrated`. In this config we make use of the [`branch`](https://docs.redpanda.com/connect/components/processors/branch/) processor as it allows us to reduce documents into smaller maps for caching and gives us greater control over how results are mapped back into the document. ```yaml input: kafka: addresses: [ TODO ] topics: [ comments ] consumer_group: benthos_comments_group pipeline: processors: # Perform both hydration and caching within a for_each block as this ensures # that a given message of a batch is cached before the next message is # hydrated, ensuring that when a message of the batch has a parent within # the same batch hydration can still work. - for_each: # Attempt to obtain parent event from cache (if the ID exists). - branch: request_map: 'root = this.comment.parent_id | deleted()' processors: - cache: operator: get resource: hydration_cache key: '${!content()}' # And if successful copy it into the field `article`. result_map: 'root.article = this.article' # Reduce comment into only fields we wish to cache. - branch: request_map: | root.comment.id = this.comment.id root.article = this.article processors: # Store reduced comment into our cache. - cache: operator: set resource: hydration_cache key: '${!json("comment.id")}' value: '${!content()}' # No `result_map` since we don't need to map into the original message. # Send resulting documents to our hydrated topic. output: kafka: addresses: [ TODO ] topic: comments_hydrated cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` This pipeline satisfies our basic needs but errors aren’t handled at all, meaning intermittent cache connectivity problems that span beyond our cache retries will result in failed documents entering our `comments_hydrated` topic. This is also the case if a comment arrives in our pipeline before its parent. There are [many patterns for error handling](https://docs.redpanda.com/connect/configuration/error_handling/) to choose from in Redpanda Connect. In this example we’re going to introduce a delayed retry queue as it enables us to reprocess failed documents after a grace period, which is isolated from our main pipeline. ## [](#adding-a-retry-queue)Adding a retry queue Our retry queue is going to be another topic called `comments_retried`. Since most errors are related to time we will delay retry attempts by storing the current timestamp after a failed request as a metadata field. We will use an input [`broker`](https://docs.redpanda.com/connect/components/inputs/broker/) so that we can consume both the `comments` and `comments_retry` topics in the same pipeline. Our config (omitting the caching sections for brevity) now looks like this: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ comments ] consumer_group: benthos_comments_group - kafka: addresses: [ TODO ] topics: [ comments_retry ] consumer_group: benthos_comments_group processors: - for_each: # Calculate time until next retry attempt and sleep for that duration. # This sleep blocks the topic 'comments_retry' but NOT 'comments', # because both topics are consumed independently and these processors # only apply to the 'comments_retry' input. - sleep: duration: '${! 3600 - ( timestamp_unix() - meta("last_attempted").number() ) }s' pipeline: processors: - try: - for_each: # Attempt to obtain parent event from cache. - branch: {} # Omitted # Reduce document into only fields we wish to cache. - branch: {} # Omitted # If we've reached this point then both processors succeeded. - mapping: 'meta output_topic = "comments_hydrated"' - catch: # If we reach here then a processing stage failed. - mapping: | meta output_topic = "comments_retry" meta last_attempted = timestamp_unix() # Send resulting documents either to our hydrated topic or the retry topic. output: kafka: addresses: [ TODO ] topic: '${!meta("output_topic")}' cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` You can find a full example [in the project repo](https://github.com/redpanda-data/connect/blob/master/config/examples/joining_streams.yaml), and with this config we can deploy as many instances of Redpanda Connect as we need as the partitions will be balanced across the consumers. --- # Page 357: Message Routing Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/message_routing.md --- # Message Routing Patterns --- title: Message Routing Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: message_routing page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: message_routing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/message_routing.adoc description: Learn how to route, filter, and discard messages in Redpanda Connect pipelines. page-topic-type: cookbook personas: streaming_developer, app_developer learning-objective-1: Implement message routing and filtering patterns in pipelines learning-objective-2: Configure conditional routing using switch and broker outputs learning-objective-3: Build dead letter queue patterns with fallback outputs page-git-created-date: "2026-04-20" page-git-modified-date: "2026-04-20" --- This cookbook demonstrates common patterns for routing and filtering messages in Redpanda Connect [pipelines](#pipeline). Use this cookbook to: - Implement message routing and filtering patterns in pipelines - Configure conditional routing using switch and broker outputs - Build dead letter queue patterns with fallback outputs ## [](#discarding-messages)Discarding messages There are two primary ways to discard messages in Redpanda Connect: using the [`drop` output](https://docs.redpanda.com/connect/components/outputs/drop/) or the [`deleted()`](https://docs.redpanda.com/connect/guides/bloblang/walkthrough/#deletions) function in a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/). ### [](#using-deleted-in-mapping)Using `deleted()` in mapping Use `deleted()` to filter messages early in the pipeline, before they reach the output stage. This is more efficient when you don’t need conditional output routing: ```yaml input: kafka: addresses: ["kafka:9092"] topics: ["events"] pipeline: processors: - mapping: | # Delete debug messages early in the pipeline root = if this.type == "debug" { deleted() } output: kafka: addresses: ["kafka:9092"] topic: production_events ``` For more filtering patterns with `deleted()`, see the [Filtering and Sampling](https://docs.redpanda.com/connect/cookbooks/filtering/) cookbook. ### [](#using-drop-output)Using `drop` output Use the [`drop` output](https://docs.redpanda.com/connect/components/outputs/drop/) to discard messages at the output stage, typically in conditional routing or as a fallback. This is useful when you’re routing different message types to different destinations: ```yaml output: switch: cases: - check: this.type == "error" output: kafka: addresses: ["kafka:9092"] topic: errors - check: this.type == "debug" output: drop: {} # Discard debug messages - output: kafka: addresses: ["kafka:9092"] topic: events ``` The `drop` output has minimal overhead and immediately acknowledges messages, making it ideal for this use case. ## [](#conditional-routing-with-switch)Conditional routing with switch The [`switch` output](https://docs.redpanda.com/connect/components/outputs/switch/) routes messages to different outputs based on conditions. Each case is evaluated in order until a match is found: ```yaml output: switch: cases: - check: this.severity == "critical" output: label: pagerduty_alerts http_client: url: https://events.pagerduty.com/v2/enqueue verb: POST - check: this.severity == "error" output: label: error_topic kafka: addresses: ["kafka:9092"] topic: errors max_in_flight: 1 - check: this.severity == "warn" output: label: warning_topic kafka: addresses: ["kafka:9092"] topic: warnings - check: this.severity == "info" output: label: info_topic kafka: addresses: ["kafka:9092"] topic: info - check: this.severity == "debug" output: label: drop_debug drop: {} # Don't store debug logs in production ``` You can label each output for monitoring and observability. ## [](#error-handling-with-fallback)Error handling with fallback The [`fallback` output](https://docs.redpanda.com/connect/components/outputs/fallback/) tries outputs in sequence. If an output fails, it moves to the next one. This is useful for implementing dead letter queue (DLQ) patterns: ### [](#basic-dlq-pattern)Basic DLQ pattern ```yaml output: fallback: - kafka: addresses: ["kafka:9092"] topic: primary_topic max_in_flight: 1 - kafka: addresses: ["kafka:9092"] topic: dlq_topic ``` If writing to `primary_topic` fails, the message is sent to `dlq_topic` instead. ### [](#dlq-with-ultimate-fallback)DLQ with ultimate fallback Add a [`drop` output](https://docs.redpanda.com/connect/components/outputs/drop/) as the final fallback to prevent pipeline failures: ```yaml output: fallback: - kafka: addresses: ["kafka:9092"] topic: primary_topic max_in_flight: 1 - kafka: addresses: ["kafka:9092"] topic: dlq_topic - drop: {} # Last resort: drop if both outputs fail ``` This ensures messages don’t cause the pipeline to block, even if both Kafka outputs fail. ### [](#multi-tier-fallback-with-retries)Multi-tier fallback with retries Combine fallback with retry configuration for more sophisticated error handling: ```yaml output: fallback: - kafka: addresses: ["kafka:9092"] topic: primary_topic max_in_flight: 10 batching: count: 100 period: 1s - retry: max_retries: 3 backoff: initial_interval: 1s max_interval: 10s output: http_client: url: https://backup-service.example.com/events verb: POST max_in_flight: 5 - kafka: addresses: ["kafka:9092"] topic: dlq_topic - drop: {} ``` This configuration: 1. Tries to write to Kafka with batching 2. Falls back to an HTTP service with retries 3. Falls back to a DLQ topic 4. Ultimately drops messages if all else fails ## [](#routing-with-broker)Routing with broker The [`broker` output](https://docs.redpanda.com/connect/components/outputs/broker/) sends messages to multiple outputs simultaneously. Unlike `switch`, all outputs receive every message: ```yaml output: broker: pattern: fan_out outputs: - kafka: addresses: ["kafka:9092"] topic: events - elasticsearch_v8: urls: ["http://localhost:9200"] index: events id: ${!json("id")} action: index - http_client: url: https://metrics.example.com/ingest verb: POST ``` This sends each message to Kafka, Elasticsearch, and an HTTP endpoint in parallel. ## [](#combining-patterns)Combining patterns You can combine these patterns for sophisticated routing logic: ```yaml output: switch: cases: # Critical events: send to multiple destinations - check: this.severity == "critical" output: broker: pattern: fan_out outputs: - kafka: addresses: ["kafka:9092"] topic: critical_events - http_client: url: https://pagerduty.example.com/alert verb: POST # Errors: send to Kafka with DLQ fallback - check: this.severity == "error" output: fallback: - kafka: addresses: ["kafka:9092"] topic: errors - kafka: addresses: ["kafka:9092"] topic: dlq # Info: send to Kafka only - check: this.severity == "info" output: kafka: addresses: ["kafka:9092"] topic: info # Debug: drop in production - check: this.severity == "debug" output: drop: {} ``` ## [](#performance-considerations)Performance considerations When choosing between routing approaches, consider: ### [](#filtering-with-deleted-vs-drop-output)Filtering with `deleted()` vs. `drop` output - **`deleted()` in mapping**: Filters messages earlier in the pipeline, reducing downstream processing - **`drop` output**: Filters at output stage, useful for conditional routing but messages still flow through processors For pure filtering without routing, use `deleted()`: ```yaml # More efficient - filters early pipeline: processors: - mapping: | root = if this.debug { deleted() } - log: {} # Doesn't process debug messages output: kafka: addresses: ["kafka:9092"] topic: events ``` For conditional routing, use `drop` output: ```yaml # Better for routing - processes all messages, routes selectively pipeline: processors: - mapping: | root.enriched = true - log: {} # Processes all messages output: switch: cases: - check: this.important output: kafka: addresses: ["kafka:9092"] topic: important - output: drop: {} # Discard non-important messages ``` ### [](#output-parallelism)Output parallelism Choose between these output types based on whether you need single-destination routing, multi-destination fan-out, or fallback behavior: - **`switch`**: Sends to one output (sequential evaluation) - **`broker`**: Sends to multiple outputs in parallel - **`fallback`**: Tries outputs sequentially until one succeeds ## [](#testing-and-debugging)Testing and debugging Use [`drop`](https://docs.redpanda.com/connect/components/outputs/drop/) to test input throughput without output bottlenecks: ```yaml input: kafka: addresses: ["kafka:9092"] topics: ["test"] consumer_group: throughput_test output: drop: {} # Measure input consumption speed ``` This reveals the maximum speed your input can consume data, isolated from output performance. ## [](#next-steps)Next steps - [Filtering and Sampling](https://docs.redpanda.com/connect/cookbooks/filtering/) - More patterns for using `deleted()` - [switch output](https://docs.redpanda.com/connect/components/outputs/switch/) - Conditional routing reference - [fallback output](https://docs.redpanda.com/connect/components/outputs/fallback/) - Error handling reference - [broker output](https://docs.redpanda.com/connect/components/outputs/broker/) - Fan-out patterns reference - [drop output](https://docs.redpanda.com/connect/components/outputs/drop/) - Drop output reference - [mapping processor](https://docs.redpanda.com/connect/components/processors/mapping/) - Bloblang mapping reference --- # Page 358: SQL Server CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/microsoft_sql_server_cdc.md --- # SQL Server CDC Patterns --- title: SQL Server CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: microsoft_sql_server_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: microsoft_sql_server_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/microsoft_sql_server_cdc.adoc description: Learn how to capture, filter, transform, and route Microsoft SQL Server change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing SQL Server CDC events learning-objective-2: Adapt integration patterns to route CDC data to Redpanda, S3, and other destinations learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-06-17" page-git-modified-date: "2026-10-14" --- The `microsoft_sql_server_cdc` input captures row-level changes from SQL Server tables using SQL Server’s built-in CDC change tables. Use these patterns to filter, transform, and route SQL Server CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing SQL Server CDC events - Adapt integration patterns to route CDC data to Redpanda, S3, and other destinations - Identify patterns for filtering and transforming change events > 📝 **NOTE** > > The `microsoft_sql_server_cdc` input is in beta. The API is subject to change. ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [rpk installation](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#sql-server-cdc)SQL Server CDC CDC must be enabled on both the database and each table you want to capture. Run the following as a user with `db_owner` or `sysadmin` privileges: ```sql -- Enable CDC on the database EXEC sys.sp_cdc_enable_db; -- Enable CDC on a table EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'orders', @role_name = NULL; ``` For cloud-managed SQL Server, see: - [Azure SQL Database CDC](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview) - [SQL Server CDC documentation](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server) ### [](#checkpoint-storage)Checkpoint storage By default, the `microsoft_sql_server_cdc` input creates a table (`rpcn.CdcCheckpointCache`) and stored procedure in your SQL Server database to store the last processed Log Sequence Number (LSN). The pipeline user must have `CREATE TABLE` and `CREATE PROCEDURE` permissions. To use an external cache instead, set the `checkpoint_cache` field to a cache label defined in `cache_resources`. See the [connector reference](https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc/#checkpoint_cache_table_name) for details. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export MSSQL_CONN="sqlserver://user:password@localhost:1433?database=mydb" (1) export REDPANDA_BROKERS=localhost:9092 (2) export S3_BUCKET=cdc-archive (3) ``` | 1 | The SQL Server connection string in sqlserver://user:password@host:port?database=dbname format. | | --- | --- | | 2 | The Redpanda broker addresses (for Redpanda output examples). | | 3 | The Amazon S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from SQL Server tables and outputs them with metadata: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.schema = meta("database_schema") root.table = meta("table") root.lsn = meta("lsn") root.data = this root.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available metadata fields, see the [metadata](https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc/#metadata) section in the connector reference. > 📝 **NOTE** > > SQL Server CDC generates two events for each row update: `update_before` (the previous row values) and `update_after` (the new row values). Most downstream use cases only need the `update_after` event. ## [](#filter-cdc-events)Filter CDC events Filter events to process only the current row state after each change, using the `operation` metadata field: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders stream_snapshot: true pipeline: processors: - mapping: | root = if meta("operation") != "insert" && meta("operation") != "update_after" { deleted() } - mapping: | root.operation = meta("operation") root.table = meta("table") root.data = this root.timestamp = now() output: stdout: codec: lines ``` This pattern: - Keeps only `insert` and `update_after` operations, dropping `delete`, `update_before`, and `read` events - Transforms the event to a simplified format with a timestamp ## [](#route-to-redpanda)Route to Redpanda Stream SQL Server changes to Redpanda for real-time processing: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders - dbo.customers stream_snapshot: true pipeline: processors: - mapping: | meta topic = meta("database_schema") + "." + meta("table") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } key: ${! json("id") } batching: count: 100 period: 1s ``` This pattern: - Uses `database_schema.table` as the Redpanda topic name - Batches messages for efficient delivery - Sets the message key to the primary key field (update `json("id")` to match your table’s primary key column) ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.schema = meta("database_schema") root.table = meta("table") root.data = this root.timestamp = now() output: aws_s3: bucket: ${S3_BUCKET} path: >- cdc/${! meta("database_schema") }/${! meta("table") }/${! timestamp_unix().format_timestamp("2006/01/02/15") }/${! uuid_v4() }.ndjson batching: count: 1000 period: 5m processors: - archive: format: lines ``` This pattern: - Organizes files by database schema, table, and time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders stream_snapshot: true output: switch: cases: - check: meta("operation") == "insert" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: sqlserver.orders.inserts - output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: sqlserver.orders.changes ``` This pattern: - Routes `insert` events to `sqlserver.orders.inserts`, and `update_before`, `update_after`, and `delete` events to `sqlserver.orders.changes` - Supports specialized downstream consumers per operation type ## [](#configure-replication-mode)Configure replication mode The `microsoft_sql_server_cdc` input supports two replication modes controlled by the `stream_snapshot` field: - `stream_snapshot: true`: Captures a full snapshot of existing table data before streaming live changes. Use this when you need a complete initial load. - `stream_snapshot: false`: Skips the snapshot and streams only changes from the current LSN position. Use this when you only need new changes going forward. ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.orders - dbo.customers stream_snapshot: true max_parallel_snapshot_tables: 4 (1) snapshot_max_batch_size: 1000 (2) ``` | 1 | Number of tables to snapshot in parallel. | | --- | --- | | 2 | Number of rows to read per batch during snapshot processing. | ## [](#filter-tables-with-patterns)Filter tables with patterns The `include` field accepts regular expressions, so you can capture multiple tables without listing each one: ```yaml input: microsoft_sql_server_cdc: connection_string: ${MSSQL_CONN} include: - dbo.* (1) exclude: - dbo.audit_log (2) stream_snapshot: false ``` | 1 | Captures all tables in the dbo schema. | | --- | --- | | 2 | Excludes a specific table matched by the include pattern. | ## [](#troubleshoot-common-issues)Troubleshoot common issues Use these steps to diagnose and fix the most common problems with the `microsoft_sql_server_cdc` input. ### [](#no-events-received)No events received If no events arrive: 1. Verify CDC is enabled on the database and table: ```sql SELECT name, is_cdc_enabled FROM sys.databases WHERE name = 'mydb'; SELECT name, is_tracked_by_cdc FROM sys.tables WHERE name = 'orders'; ``` 2. Confirm CDC agent jobs are running: ```sql EXEC sys.sp_cdc_help_jobs; ``` 3. Check that the table pattern in `include` matches the target table, using `schema.table` format. ### [](#checkpoint-table-creation-fails)Checkpoint table creation fails If the pipeline fails to start because it cannot create the checkpoint table: - Grant `CREATE TABLE` and `CREATE PROCEDURE` permissions to the pipeline user, or - Create the schema manually (`CREATE SCHEMA rpcn`) and grant the user permissions on it, or - Use an external `checkpoint_cache` to avoid creating a table in SQL Server. ### [](#duplicate-events)Duplicate events The `microsoft_sql_server_cdc` input provides at-least-once delivery. If the pipeline fails between checkpoints, events may be re-read on restart. To handle duplicates: - Use idempotent processing in downstream systems. - Deduplicate using the `lsn` metadata field. - Lower `checkpoint_limit` to reduce the window of possible duplicates. ## [](#next-steps)Next steps - [SQL Server CDC input reference](https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) - [Register CDC schemas in a schema registry](https://docs.redpanda.com/connect/cookbooks/cdc_schema_registry/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 359: MongoDB CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/mongodb_cdc.md --- # MongoDB CDC Patterns --- title: MongoDB CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: mongodb_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mongodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/mongodb_cdc.adoc description: Learn how to capture, filter, transform, and route MongoDB change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing MongoDB CDC events learning-objective-2: Adapt patterns to route CDC data to Redpanda, S3, and other destinations learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-06-17" page-git-modified-date: "2026-10-14" --- The `mongodb_cdc` input captures document-level changes from MongoDB collections using change streams. Use these patterns to filter, transform, and route MongoDB CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing MongoDB CDC events - Adapt patterns to route CDC data to Redpanda, S3, and other destinations - Identify patterns for filtering and transforming change events ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [rpk installation](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#mongodb-replica-set)MongoDB replica set The `mongodb_cdc` input requires a MongoDB database running as a replica set or in a sharded cluster. Change streams are not available on standalone MongoDB instances. To verify your deployment is running as a replica set: ```javascript rs.status() ``` For cloud-managed MongoDB, [MongoDB Atlas](https://www.mongodb.com/docs/atlas/) uses replica sets by default. ### [](#checkpoint-cache)Checkpoint cache The `mongodb_cdc` input requires a [cache](https://docs.redpanda.com/connect/components/caches/about/) to store the oplog position between restarts. The examples in this cookbook use a Redis cache: ```bash export REDIS_URL=redis://localhost:6379 ``` For production use, any persistent cache supported by Redpanda Connect works, such as `aws_dynamodb` or `postgres`. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export MONGODB_URL="mongodb://user:password@localhost:27017" (1) export MONGODB_DB="mydb" (2) export REDIS_URL=redis://localhost:6379 (3) export REDPANDA_BROKERS=localhost:9092 (4) export S3_BUCKET=cdc-archive (5) ``` | 1 | The MongoDB connection string. | | --- | --- | | 2 | The name of the database to capture changes from. | | 3 | The Redis URL for checkpoint storage. | | 4 | The Redpanda broker addresses (for Redpanda output examples). | | 5 | The S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from MongoDB collections and outputs them with metadata: ```yaml input: mongodb_cdc: url: ${MONGODB_URL} database: ${MONGODB_DB} collections: - orders checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.collection = meta("collection") root.operation_time = meta("operation_time") root.data = this root.timestamp = now() output: stdout: codec: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` For details on the CDC event message structure and available metadata fields, see the [metadata](https://docs.redpanda.com/connect/components/inputs/mongodb_cdc/#metadata) section in the connector reference. ## [](#filter-cdc-events)Filter CDC events Filter events to process only specific change types using the `operation` metadata field: ```yaml input: mongodb_cdc: url: ${MONGODB_URL} database: ${MONGODB_DB} collections: - orders checkpoint_cache: redis_cache stream_snapshot: false pipeline: processors: - mapping: | root = if meta("operation") == "delete" || meta("operation") == "read" { deleted() } - mapping: | root.operation = meta("operation") root.collection = meta("collection") root.data = this root.timestamp = now() output: stdout: codec: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Filters to only `insert`, `update`, and `replace` operations, dropping `delete` and `read` events - Transforms the event to a simplified format with a timestamp ## [](#route-to-redpanda)Route to Redpanda Stream MongoDB changes to Redpanda for real-time processing: ```yaml input: mongodb_cdc: url: ${MONGODB_URL} database: ${MONGODB_DB} collections: - orders - customers checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | meta topic = meta("collection") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } key: ${! json("_id") } batching: count: 100 period: 1s cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Uses the collection name as the Redpanda topic - Batches messages for efficient delivery - Sets the message key to the document’s `_id` field ## [](#route-to-s3)Route to S3 Archive CDC events to Amazon S3 for long-term storage and analytics: ```yaml input: mongodb_cdc: url: ${MONGODB_URL} database: ${MONGODB_DB} collections: - orders checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.collection = meta("collection") root.data = this root.timestamp = now() output: aws_s3: bucket: ${S3_BUCKET} path: >- cdc/${! meta("collection") }/${! timestamp_unix().format_timestamp("2006/01/02/15") }/${! uuid_v4() }.ndjson batching: count: 1000 period: 5m processors: - archive: format: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Organizes files by collection and time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: mongodb_cdc: url: ${MONGODB_URL} database: ${MONGODB_DB} collections: - orders checkpoint_cache: redis_cache stream_snapshot: false output: switch: cases: - check: meta("operation") == "insert" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: mongodb.orders.inserts - output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: mongodb.orders.changes cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Routes `insert` events to one Redpanda topic and all other change events to another - Supports specialized downstream consumers per operation type ## [](#configure-replication-mode)Configure replication mode The `mongodb_cdc` input supports two replication modes controlled by the `stream_snapshot` field: - `stream_snapshot: true`: Captures a full snapshot of existing collection data before streaming live changes. Use this when you need a complete initial load. - `stream_snapshot: false`: Skips the snapshot and streams only changes from the current oplog position. Use this when you only need new changes going forward. ```yaml input: mongodb_cdc: url: ${MONGODB_URL} database: ${MONGODB_DB} collections: - orders checkpoint_cache: redis_cache stream_snapshot: true snapshot_parallelism: 4 (1) cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` | 1 | Number of parallel connections to use when reading the snapshot. | | --- | --- | > 📝 **NOTE** > > If the pipeline restarts during a snapshot, Redpanda Connect must restart the snapshot from the beginning to record a fresh oplog position in the checkpoint cache. ## [](#troubleshoot-common-issues)Troubleshoot common issues Use these steps to diagnose and fix the most common problems with the `mongodb_cdc` input. ### [](#no-events-received)No events received If no events arrive: 1. Verify the MongoDB deployment is a replica set: ```javascript rs.status() ``` 2. Grant the required privileges to the user: ```javascript db.grantRolesToUser("your_user", [ { role: "read", db: "mydb" }, { role: "clusterMonitor", db: "admin" } ]) ``` 3. Check that the collection names in `collections` exist in the specified `database`. ### [](#pipeline-restarts-lose-position)Pipeline restarts lose position If the pipeline restarts and replays events from the beginning: - Verify the checkpoint cache is persistent and accessible. - Check that `checkpoint_key` is consistent across restarts (default: `mongodb_cdc_checkpoint`). - Use a durable cache backend such as Redis with persistence enabled, `aws_dynamodb`, or `postgres`. ### [](#duplicate-events)Duplicate events The `mongodb_cdc` input provides at-least-once delivery. If the pipeline fails between checkpoints, events may be re-read on restart. To handle duplicates: - Use idempotent processing in downstream systems. - Deduplicate using the `operation_time` metadata field. - Lower `checkpoint_limit` to reduce the window of possible duplicates. ## [](#next-steps)Next steps - [MongoDB CDC input reference](https://docs.redpanda.com/connect/components/inputs/mongodb_cdc/) - [Redis cache](https://docs.redpanda.com/connect/components/caches/redis/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 360: MySQL CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/mysql_cdc.md --- # MySQL CDC Patterns --- title: MySQL CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: mysql_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mysql_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/mysql_cdc.adoc description: Learn how to capture, filter, transform, and route MySQL change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing MySQL CDC events learning-objective-2: Adapt integration patterns to route CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events learning-objective-4: Plan inputs and pipelines that capture changes from many MySQL tables page-git-created-date: "2026-06-17" page-git-modified-date: "2026-10-14" --- The `mysql_cdc` input captures row-level changes from MySQL tables using the binary log (binlog). Use these patterns to filter, transform, and route MySQL CDC events to Redpanda, S3, and other destinations, and to plan captures that span many tables. Use this cookbook to: - Apply reusable patterns for capturing MySQL CDC events - Adapt integration patterns to route CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events - Plan inputs and pipelines that capture changes from many MySQL tables ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [rpk installation](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#mysql-binlog)MySQL binlog The source MySQL database must have binary logging enabled with row-based replication: ```sql -- Verify binary logging is enabled SHOW VARIABLES LIKE 'log_bin'; -- Verify row-based format SHOW VARIABLES LIKE 'binlog_format'; ``` Set `log_bin` to `ON` and `binlog_format` to `ROW`. For cloud-managed MySQL, see: - [Change data capture on Amazon RDS for MySQL](https://aws.amazon.com/blogs/database/enable-change-data-capture-on-amazon-rds-for-mysql-applications-that-are-using-xa-transactions/) - [Azure MySQL Database CDC](https://learn.microsoft.com/en-us/fabric/real-time-hub/add-source-mysql-database-cdc) - [Google Cloud SQL for MySQL](https://cloud.google.com/datastream/docs/configure-cloudsql-mysql) ### [](#checkpoint-cache)Checkpoint cache The `mysql_cdc` input requires a [cache](https://docs.redpanda.com/connect/components/caches/about/) to store the binlog position between restarts. The examples in this cookbook use a Redis cache: ```bash export REDIS_URL=redis://localhost:6379 ``` For production use, any persistent cache supported by Redpanda Connect works, such as `aws_dynamodb` or `postgres`. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export MYSQL_DSN="user:password@tcp(localhost:3306)/mydb" (1) export MYSQL_TABLES="orders,customers" (2) export REDIS_URL=redis://localhost:6379 (3) export REDPANDA_BROKERS=localhost:9092 (4) export S3_BUCKET=cdc-archive (5) ``` | 1 | The MySQL DSN in user:password@tcp(host:port)/database format. | | --- | --- | | 2 | Comma-separated list of tables to capture. List table names without a database prefix: the input reads tables from the database named in the DSN. | | 3 | The Redis URL for checkpoint storage. | | 4 | The Redpanda broker addresses (for Redpanda output examples). | | 5 | The S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from MySQL tables and outputs them with metadata: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.table = meta("table") root.binlog_position = meta("binlog_position") root.data = this root.timestamp = now() output: stdout: codec: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` For details on the CDC event message structure and available metadata fields, see the [metadata](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/#metadata) section in the connector reference. ## [](#filter-cdc-events)Filter CDC events Filter events to process only specific change types using the `operation` metadata field: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | # Drop delete events, pass through inserts and updates root = if meta("operation") == "delete" { deleted() } else { { "operation": meta("operation"), "table": meta("table"), "data": this, "timestamp": now() } } output: stdout: codec: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Filters out `delete` events, passing through `insert` and `update` operations - Transforms the event to a simplified format with a timestamp ## [](#route-to-redpanda)Route to Redpanda Stream MySQL changes to Redpanda for real-time processing: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders - customers checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | root = this meta topic = meta("table") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } key: ${! json("id") } batching: count: 100 period: 1s cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Uses the table name as the Redpanda topic - Batches messages for efficient delivery - Sets the message key to the row’s primary key field (update `json("id")` to match your table’s primary key column) ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders checkpoint_cache: redis_cache stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.table = meta("table") root.data = this root.timestamp = now() output: aws_s3: bucket: ${S3_BUCKET} path: >- cdc/${! meta("table") }/${! timestamp_unix().format_timestamp("2006/01/02/15") }/${! uuid_v4() }.ndjson batching: count: 1000 period: 5m processors: - archive: format: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Organizes files by table and time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders checkpoint_cache: redis_cache stream_snapshot: true output: switch: cases: - check: meta("operation") == "insert" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: mysql.orders.inserts - output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: mysql.orders.changes cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Routes `insert` events to one Redpanda topic and all other change events to another - Supports specialized downstream consumers per operation type ## [](#configure-replication-mode)Configure replication mode The `mysql_cdc` input supports two replication modes controlled by the `stream_snapshot` field: - `stream_snapshot: true`: Captures a full snapshot of existing table data before streaming live changes. Use this when you need a complete initial load. - `stream_snapshot: false`: Skips the snapshot and streams only changes from the current binlog position. Use this when you only need new changes going forward. ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: - orders checkpoint_cache: redis_cache stream_snapshot: true snapshot_max_batch_size: 1000 (1) cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` | 1 | Number of rows to read per batch during snapshot processing. | | --- | --- | ## [](#capture-many-tables)Capture many tables To capture changes from dozens or hundreds of tables, group many tables into each `mysql_cdc` input instead of creating one input or pipeline for each table. Fewer inputs mean fewer binlog connections to your MySQL server and fewer pipelines to run. ### [](#how-inputs-read-the-binlog)How inputs read the binlog MySQL writes changes for every table on the server to a single binary log. Each `mysql_cdc` input opens its own binlog connection, and the server sends that connection the full binlog stream. The input then discards events for tables that are not in its `tables` list. As a result, every additional input reads and transfers the same binlog again. If you create one pipeline per table for 100 tables, the server streams the full binlog 100 times, and each pipeline discards almost all of it. A single input that lists all 100 tables reads the binlog once. ### [](#group-tables-into-lanes)Group tables into lanes A lane (sometimes called a swimlane) is one `mysql_cdc` input that reads the binlog once for a group of tables. Start with as few lanes as possible, and add a lane only when you have a reason to separate tables: - Tables in different databases. Each input reads tables from the database named in its `dsn`, so tables in another database need their own input. - Very large tables whose snapshot you want to run, restart, or tune separately from the rest. See [Plan snapshots for large lanes](#plan-snapshots-for-large-lanes). - Groups of tables that you want to deploy, stop, or change independently, such as tables owned by different teams. Each lane stores its own binlog position. If lanes share a checkpoint cache, give each lane a unique [`checkpoint_key`](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/#checkpoint_key). Lanes that use the same key overwrite each other’s position. The `mysql_cdc` input has no setting for a replica server ID. The following lane captures six tables from one database and writes each table to its own Redpanda topic: ```yaml input: mysql_cdc: dsn: ${MYSQL_DSN} tables: (1) - orders - order_items - customers - addresses - products - inventory checkpoint_cache: redis_cache checkpoint_key: mysql_binlog_position_sales (2) stream_snapshot: true max_parallel_snapshot_tables: 4 (3) pipeline: processors: - mapping: | root = this meta topic = "mysql." + meta("table") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } batching: count: 100 period: 1s cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` | 1 | List every table in the lane. Use table names only, without a database prefix. | | --- | --- | | 2 | A checkpoint key that is unique to this lane. | | 3 | The number of tables to snapshot in parallel. The default is 1. | The mapping sets the topic from the `table` metadata field, so one output serves every table in the lane. ### [](#plan-snapshots-for-large-lanes)Plan snapshots for large lanes When `stream_snapshot` is `true` and the checkpoint cache holds no binlog position for the lane, the input snapshots every table in its `tables` list before it streams changes. Keep the following behavior in mind when you size a lane: - The input runs `FLUSH TABLES …​ WITH READ LOCK` on all tables in the lane, which blocks writes to them until the input has opened its snapshot transactions and read the binlog position. - The input stores the binlog position only after the whole snapshot is acknowledged. If the pipeline restarts before then, the snapshot of every table in the lane starts again. - After the snapshot, the input streams changes from the binlog position that it read before the snapshot. Your MySQL server must still have the binlog file for that position when the snapshot finishes, so set the binlog retention period on the server to cover the full snapshot time. - Use [`max_parallel_snapshot_tables`](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/#max_parallel_snapshot_tables) to snapshot several tables at the same time. Each parallel snapshot worker holds its own transaction on a separate MySQL connection. ### [](#add-tables-to-an-existing-lane)Add tables to an existing lane If you add a table to the `tables` list of a lane that already has a stored binlog position, the input does not snapshot the new table. It streams changes to that table from the stored position onward, but it does not read rows that existed before. To load the existing rows of a new table, capture it in a new lane with its own `checkpoint_key` and `stream_snapshot: true`, instead of adding it to an existing lane. ### [](#pipeline-limits-in-redpanda-cloud)Pipeline limits in Redpanda Cloud In Redpanda Cloud, each cluster can create up to 100 pipelines. To run more, contact Redpanda support. A design with one pipeline per table can reach this limit quickly. For details, see [Redpanda Connect quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/). ### [](#planning-checklist)Planning checklist Before you build pipelines for a large migration, answer these questions: - How many tables do you need to capture, and in how many databases? - How many lanes do you need? Start with one lane per database, and split only for the reasons in [Group tables into lanes](#group-tables-into-lanes). - Does each lane have a unique `checkpoint_key`, or its own checkpoint cache? - How many rows does each lane need to snapshot, and how long does each snapshot take? - Does the binlog retention period on your MySQL server cover the longest snapshot? - Do all tables that you snapshot have a primary key? Snapshots require one. - How often do you plan to add tables, and how do you load their existing rows? - In Redpanda Cloud, how many pipelines does the cluster run in total, including pipelines for other workloads? ## [](#troubleshoot-common-issues)Troubleshoot common issues Use these steps to diagnose and fix the most common problems with the `mysql_cdc` input. ### [](#no-events-received)No events received If you’re not receiving events: 1. Verify binary logging is enabled: ```sql SHOW VARIABLES LIKE 'log_bin'; SHOW VARIABLES LIKE 'binlog_format'; ``` 2. Confirm the user has the required MySQL privileges: ```sql GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'your_user'@'%'; GRANT SELECT ON mydb.* TO 'your_user'@'%'; ``` 3. Check that the names in `tables` are table names only, such as `orders`, not `mydb.orders`. The input reads tables from the database named in `dsn`. ### [](#pipeline-restarts-lose-position)Pipeline restarts lose position If the pipeline restarts and replays events from the beginning: - Verify the checkpoint cache is persistent and accessible. - Check that `checkpoint_key` is consistent across restarts (default: `mysql_binlog_position`). - Use a durable cache backend such as Redis with persistence enabled, `aws_dynamodb`, or `postgres`. ### [](#duplicate-events)Duplicate events The `mysql_cdc` input provides at-least-once delivery. If the pipeline fails between checkpoints, events may be re-read on restart. To handle duplicates: - Use idempotent processing in downstream systems. - Deduplicate using the `binlog_position` metadata field. - Lower `checkpoint_limit` to reduce the window of possible duplicates. ## [](#next-steps)Next steps - [MySQL CDC input reference](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/) - [Redis cache](https://docs.redpanda.com/connect/components/caches/redis/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) - [Register CDC schemas in a schema registry](https://docs.redpanda.com/connect/cookbooks/cdc_schema_registry/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 361: Oracle Database CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/oracledb_cdc.md --- # Oracle Database CDC Patterns --- title: Oracle Database CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: oracledb_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: oracledb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/oracledb_cdc.adoc description: Learn how to capture, filter, transform, and route Oracle Database change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Find reusable patterns for capturing Oracle Database CDC events learning-objective-2: Look up integration patterns for routing CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-04-10" page-git-modified-date: "2026-10-14" --- The Oracle Database CDC input enables capturing row-level changes from Oracle tables using LogMiner. This cookbook provides reusable patterns for filtering, transforming, and routing Oracle CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Find reusable patterns for capturing Oracle Database CDC events - Look up integration patterns for routing CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events ## [](#prerequisites)Prerequisites Before using these patterns, ensure you have the following configured: ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#oracle-database-setup)Oracle Database setup The Oracle database must be configured for CDC: #### [](#enable-supplemental-logging)Enable supplemental logging Supplemental logging must be enabled at the database and table level: ```sql -- Enable database-level supplemental logging ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; -- Enable table-level supplemental logging (for each table) ALTER TABLE my_schema.my_table ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ``` #### [](#enable-archivelog-mode)Enable archivelog mode The database must be in ARCHIVELOG mode for LogMiner to access redo logs: ```sql -- Check if archivelog is enabled ARCHIVE LOG LIST; -- Enable archivelog mode (if needed) SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE ARCHIVELOG; ALTER DATABASE OPEN; ``` #### [](#grant-permissions)Grant permissions The database user needs specific permissions to read redo logs: ```sql -- Create user for CDC CREATE USER cdc_user IDENTIFIED BY ; -- Grant necessary permissions GRANT CREATE SESSION TO cdc_user; GRANT SELECT ANY TABLE TO cdc_user; GRANT SELECT_CATALOG_ROLE TO cdc_user; GRANT EXECUTE_CATALOG_ROLE TO cdc_user; GRANT SELECT ANY TRANSACTION TO cdc_user; GRANT LOGMINING TO cdc_user; -- Grant access to specific tables GRANT SELECT ON my_schema.my_table TO cdc_user; -- Create schema for checkpoint table CREATE USER rpcn IDENTIFIED BY ; GRANT CREATE TABLE TO rpcn; GRANT CREATE PROCEDURE TO rpcn; GRANT UNLIMITED TABLESPACE TO rpcn; ``` ### [](#oracle-wallet-for-ssl-connections)Oracle Wallet (for SSL connections) For secure connections using Oracle Wallet: 1. Create a wallet directory: ```bash mkdir -p /opt/oracle/wallet ``` 2. Use Oracle Wallet Manager or `orapki` to create and configure the wallet: ```bash orapki wallet create -wallet /opt/oracle/wallet -auto_login orapki wallet add -wallet /opt/oracle/wallet -trusted_cert -cert ca.crt orapki wallet add -wallet /opt/oracle/wallet -user_cert -cert client.crt -pwd ``` 3. The wallet directory should contain either: - `cwallet.sso` (auto-login wallet, no password required) - `ewallet.p12` (requires password) ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for Oracle configuration. This allows you to keep credentials secure and separate from your pipeline configuration files. ```bash export ORACLE_CONNECTION_STRING=oracle://cdc_user:password@host:1521/service_name (1) export ORACLE_WALLET_PATH=/opt/oracle/wallet (2) export ORACLE_WALLET_PASSWORD=wallet_password (3) export REDPANDA_BROKERS=localhost:9092 (4) export S3_BUCKET=my-cdc-bucket (5) ``` | 1 | The Oracle connection string with credentials. | | --- | --- | | 2 | Path to Oracle Wallet directory (optional, for SSL). | | 3 | Wallet password if using ewallet.p12 (optional). | | 4 | The Redpanda broker addresses (for Redpanda output examples). | | 5 | The S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from Oracle tables and outputs them with metadata: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" - "MY_SCHEMA\\.CUSTOMERS" exclude: - "MY_SCHEMA\\..*_TEMP" pipeline: processors: # Extract the change event details - mapping: | root = this root.metadata.operation = meta("operation") root.metadata.table = meta("table_name") root.metadata.schema = meta("database_schema") root.metadata.scn = meta("scn") root.metadata.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available metadata fields, see the [metadata section](https://docs.redpanda.com/connect/components/inputs/oracledb_cdc/#metadata) in the connector reference. ## [](#filter-cdc-events)Filter CDC events You can filter events to process only specific change types: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: # Filter out delete events - mapping: | root = if meta("operation") == "delete" { deleted() } else { this } # Add operation metadata to the message - mapping: | root = this.merge({ "operation": meta("operation"), "table": meta("table_name") }) output: stdout: codec: lines ``` This example: - Filters out `delete` events using `deleted()` - Adds operation type and timestamp to the output - Transforms the event to include metadata ## [](#stream-initial-snapshot)Stream initial snapshot Capture both existing data and ongoing changes: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" stream_snapshot: true (1) snapshot_max_batch_size: 5000 (2) max_parallel_snapshot_tables: 2 (3) pipeline: processors: - mapping: | root = this # Tag snapshot vs CDC events root.event_type = if meta("operation") == "read" { "snapshot" } else { "cdc" } output: stdout: codec: lines ``` | 1 | Enable initial snapshot of existing table data. | | --- | --- | | 2 | Number of rows per batch during snapshot (higher = faster, more memory). | | 3 | Number of tables to snapshot in parallel. | > 📝 **NOTE** > > Snapshot events have `operation` metadata set to `"read"`, while CDC events use `"insert"`, `"update"`, or `"delete"`. ## [](#route-to-redpanda)Route to Redpanda Stream Oracle changes to Redpanda for real-time processing: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" - "MY_SCHEMA\\.CUSTOMERS" pipeline: processors: # Create a composite key from the primary key fields - mapping: | meta kafka_key = this.ID.string() # Add CDC metadata as message headers meta operation = meta("operation") meta table = meta("table_name") meta scn = meta("scn") output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: 'oracle-cdc-${! meta("table_name").lowercase() }' (1) max_in_flight: 10 compression: snappy batching: count: 100 period: 100ms ``` | 1 | Route to different topics based on table name. | | --- | --- | This example: - Creates a message key from the primary key for consistent partitioning - Routes each table to a separate topic - Batches messages for efficient delivery - Adds CDC metadata as Kafka headers ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: - mapping: | root = this root.operation = meta("operation") root.table = meta("table_name") root.scn = meta("scn") root.captured_at = now().ts_format("2006-01-02T15:04:05Z07:00") output: aws_s3: bucket: ${S3_BUCKET} path: 'oracle-cdc/${! meta("table_name") }/${! timestamp_unix().ts_format("2006/01/02/15") }/${! uuid_v4() }.jsonl' (1) batching: count: 1000 period: 5m processors: - archive: format: json_array (2) - compress: algorithm: gzip (3) ``` | 1 | Organize files by table and time-based partitions (year/month/day/hour). | | --- | --- | | 2 | Archive as JSON array for easier downstream parsing. | | 3 | Compress with gzip to reduce storage costs. | This example: - Organizes files by table name and hourly partitions - Batches events and archives them as compressed JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: # Add routing metadata - mapping: | root = this root.event_type = meta("operation") root.table = meta("table_name") output: switch: cases: # Route INSERT events - check: this.event_type == "insert" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: orders-inserts # Route UPDATE events - check: this.event_type == "update" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: orders-updates # Route DELETE events - check: this.event_type == "delete" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: orders-deletes # Fallback for snapshot reads - output: drop: {} ``` This pattern: - Separates processing pipelines for inserts, updates, and deletes - Routes each operation type to a dedicated topic - Enables specialized downstream consumers per operation type ## [](#detect-changed-columns)Detect changed columns For update operations, identify which columns changed: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" processors: - mapping: | # Only process update operations root = if meta("operation") != "update" { deleted() } # Extract the update event details # Oracle CDC provides before/after values for updates root.id = this.ID root.table = meta("table_name") root.operation = meta("operation") root.scn = meta("scn") # Include the full after-state root.after = this output: stdout: codec: lines ``` > 📝 **NOTE** > > Oracle CDC captures the full row state after the update. For fine-grained change detection, you may need to maintain a previous state externally or use Oracle’s built-in change tracking features. ## [](#checkpointing)Checkpointing The Oracle CDC input automatically manages checkpoints using a table in the Oracle database: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE (1) checkpoint_cache_key: orders_cdc_job (2) checkpoint_limit: 500 (3) ``` | 1 | Table name for storing checkpoint state (created automatically if it doesn’t exist). | | --- | --- | | 2 | Unique key to identify this CDC job (useful when running multiple CDC pipelines). | | 3 | Checkpoint after every 500 messages (lower = better recovery, higher = fewer writes). | The checkpoint table stores the System Change Number (SCN) of the last successfully processed event. On restart, CDC resumes from the last checkpoint. ### [](#using-an-external-cache)Using an external cache For better separation or to use a different database for checkpointing: ```yaml cache_resources: - label: postgres_cache postgres: dsn: postgres://user:pass@postgres-host:5432/checkpoints input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" checkpoint_cache: postgres_cache (1) checkpoint_cache_key: orders_cdc_job ``` | 1 | Use an external cache resource instead of the default Oracle table. | | --- | --- | ## [](#performance-tuning)Performance tuning Optimize throughput and latency with these LogMiner settings: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\..*" logminer: scn_window_size: 20000 (1) min_scn_window_size: 1000 (2) max_scn_window_size: 100000 (3) mining_interval: 100ms (4) backoff_interval: 1s (5) lob_enabled: true (6) checkpoint_limit: 2000 (7) batching: count: 100 period: 1s ``` | 1 | Number of SCNs to process in each LogMiner cycle. | | --- | --- | | 2 | Minimum number of SCNs to process in each LogMiner cycle. | | 3 | Maximum number of SCNs to process in each LogMiner cycle (higher = more memory, fewer sessions). | | 4 | How often to query LogMiner for new changes (lower = less latency, more CPU). | | 5 | Backoff when no changes are detected (reduces CPU usage during idle periods). | | 6 | Enable capture of LOB (CLOB/BLOB) columns. Set this to false if you’re streaming tables with no LOB data. | | 7 | Number of messages to process before checkpointing. | ### [](#throughput-considerations)Throughput considerations - Larger `max_scn_window_size` reduces LogMiner overhead but increases memory usage - Shorter `mining_interval` reduces latency but increases database load - Higher `checkpoint_limit` improves throughput but increases recovery time after failures - LOB columns significantly impact performance; disable `lob_enabled` if the tables you’re tracking contain no LOB/CLOB data. ### [](#dynamic-scn-windowing)Dynamic SCN windowing The Oracle CDC connector scans ranges of SCNs within the Oracle redo logs using a dynamically sized window. When a mining cycle finds more backlog than the current window covers, the window grows by `scn_window_size` each cycle, up to a ceiling of `max_scn_window_size`. Once the connector catches up to the database’s current SCN within a cycle, the window shrinks by `scn_window_size` each cycle, down to a floor of `min_scn_window_size` - note that if `scn_window_size` is larger than the distance to that floor (as it is with the default values), the window drops to the floor in a single step rather than tapering gradually. > 📝 **NOTE** > > `min_scn_window_size` has a second, independent role: if the real SCN gap between the connector’s current position and the database is smaller than this value, the mining cycle is skipped entirely and the connector backs off, avoiding excessive LogMiner start/stop cycles on low-traffic databases. ### [](#prefetch_rows-configuration)PREFETCH_ROWS configuration The Oracle driver fetches 25 rows per network round trip by default, which can make large committed transactions appear minutes late while the database, network and connector all look idle: each round trip costs a full network exchange, and a large transaction requires thousands of them. Raise the fetch size with the `PREFETCH_ROWS` query parameter on `connection_string`, for example `?PREFETCH_ROWS=1000`. Equally, on high-throughput systems increasing this value reduces network round trips, improving the overall throughput of the connector. ## [](#secure-connections-with-oracle-wallet)Secure connections with Oracle Wallet Use Oracle Wallet for SSL/TLS connections: ```yaml input: oracledb_cdc: connection_string: oracle://cdc_user:password@host:1522/service wallet_path: ${ORACLE_WALLET_PATH} (1) wallet_password: ${ORACLE_WALLET_PASSWORD} (2) include: - "MY_SCHEMA\\.ORDERS" ``` | 1 | Path to the wallet directory containing cwallet.sso or ewallet.p12. | | --- | --- | | 2 | Wallet password (only required for ewallet.p12 files). | Alternatively, specify wallet configuration in the connection string: ```yaml input: oracledb_cdc: connection_string: oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true include: - "MY_SCHEMA\\.ORDERS" ``` ## [](#troubleshoot)Troubleshoot ### [](#no-events-received)No events received If you’re not receiving events: 1. Verify supplemental logging is enabled: ```sql -- Check database-level logging SELECT SUPPLEMENTAL_LOG_DATA_MIN FROM V$DATABASE; -- Check table-level logging SELECT LOG_GROUP_NAME, TABLE_NAME, DECODE(ALWAYS, 'ALWAYS', 'Unconditional', NULL, 'Conditional') ALWAYS FROM DBA_LOG_GROUPS WHERE OWNER = 'MY_SCHEMA' AND TABLE_NAME = 'MY_TABLE'; ``` 2. Check that the database is in ARCHIVELOG mode: ```sql SELECT LOG_MODE FROM V$DATABASE; ``` 3. Verify the user has correct permissions: ```sql SELECT * FROM DBA_SYS_PRIVS WHERE GRANTEE = 'CDC_USER'; ``` 4. Make sure changes are being made to the monitored tables ### [](#slow-event-processing)Slow event processing If CDC is lagging behind: - Increase `max_scn_window_size` to let the connector mine larger windows per LogMiner cycle during backlog (see [Dynamic SCN windowing](#dynamic-scn-windowing)) - Reduce `checkpoint_limit` to checkpoint more frequently (helps during restarts) - Disable `lob_enabled` if LOB columns aren’t needed - Check Oracle alert logs for performance issues - Monitor the size of redo logs and archive logs ### [](#connection-issues-with-oracle-wallet)Connection issues with Oracle Wallet If SSL connections fail: 1. Verify wallet files exist and have correct permissions: ```bash ls -la /opt/oracle/wallet ``` 2. Check Oracle TNS configuration 3. Verify the wallet password for `ewallet.p12` files 4. Test the connection using Oracle tools (sqlplus, SQL Developer) ### [](#checkpoint-table-errors)Checkpoint table errors If you see errors related to the checkpoint table: 1. Ensure the `rpcn` schema exists: ```sql CREATE USER rpcn IDENTIFIED BY ; GRANT CREATE TABLE TO rpcn; GRANT CREATE PROCEDURE TO rpcn; GRANT UNLIMITED TABLESPACE TO rpcn; ``` 2. Verify the CDC user has access to the checkpoint table: ```sql GRANT SELECT, INSERT, UPDATE ON rpcn.cdc_checkpoint_cache TO cdc_user; ``` ## [](#suggested-reading)Suggested reading - [Oracle CDC Input Reference](https://docs.redpanda.com/connect/components/inputs/oracledb_cdc/) - [Redpanda Output](https://docs.redpanda.com/connect/components/outputs/redpanda/) - [AWS S3 Output](https://docs.redpanda.com/connect/components/outputs/aws_s3/) - [Register CDC schemas in a schema registry](https://docs.redpanda.com/connect/cookbooks/cdc_schema_registry/) - [Oracle LogMiner Documentation](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html) - [Configuring Supplemental Logging](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html#GUID-D857AF96-AC24-4CA1-B620-8EA3DF30D72E) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 362: PostgreSQL CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/postgres_cdc.md --- # PostgreSQL CDC Patterns --- title: PostgreSQL CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: postgres_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: postgres_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/postgres_cdc.adoc description: Learn how to capture, filter, transform, and route PostgreSQL change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing PostgreSQL CDC events learning-objective-2: Adapt integration patterns to route CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events learning-objective-4: Backfill and re-snapshot tables with incremental snapshots page-git-created-date: "2026-06-17" page-git-modified-date: "2026-10-14" --- The `postgres_cdc` input captures row-level changes from PostgreSQL tables using logical replication and the Write-Ahead Log (WAL). Use these patterns to filter, transform, and route PostgreSQL CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing PostgreSQL CDC events - Adapt integration patterns to route CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events - Backfill and re-snapshot tables with incremental snapshots ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [rpk installation](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) for installation instructions. ### [](#postgresql-logical-replication)PostgreSQL logical replication The source PostgreSQL database must have logical replication enabled. Verify the setting by running: ```sql SHOW wal_level; ``` The `wal_level` value must be `logical`. For instructions on enabling logical replication, see the [postgres\_cdc prerequisites](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/#prerequisites). For cloud-managed PostgreSQL, see: - [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication.html) - [Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical) - [Google Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication) ### [](#replication-slot)Replication slot The `postgres_cdc` input uses a PostgreSQL replication slot to track its position in the WAL. The slot is created automatically when the pipeline starts, using the name specified in the `slot_name` field. Each pipeline must use a unique slot name. Replication slots persist in the database, so the pipeline resumes from where it left off after a restart. ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export PG_DSN="postgres://user:password@localhost:5432/mydb" (1) export PG_SCHEMA="public" (2) export PG_SLOT_NAME="my_connect_slot" (3) export REDPANDA_BROKERS=localhost:9092 (4) export S3_BUCKET=cdc-archive (5) ``` | 1 | The PostgreSQL DSN in postgres://user:password@host:port/dbname format. | | --- | --- | | 2 | The schema containing the tables to capture. | | 3 | A unique name for the replication slot. | | 4 | The Redpanda broker addresses (for Redpanda output examples). | | 5 | The S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from PostgreSQL tables and outputs them with metadata: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders slot_name: ${PG_SLOT_NAME} stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.table = meta("table") root.lsn = meta("lsn") root.data = this root.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available metadata fields, see the [metadata](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/#metadata) section in the connector reference. ## [](#filter-cdc-events)Filter CDC events Filter events to process only specific change types using the `operation` metadata field: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders slot_name: ${PG_SLOT_NAME} stream_snapshot: false pipeline: processors: - mapping: | root = if meta("operation") != "insert" && meta("operation") != "update" { deleted() } - mapping: | root.operation = meta("operation") root.table = meta("table") root.data = this root.timestamp = now() output: stdout: codec: lines ``` This pattern: - Filters to only `insert` and `update` operations, dropping `delete` and `read` events - Transforms the event to a simplified format with a timestamp ## [](#route-to-redpanda)Route to Redpanda Stream PostgreSQL changes to Redpanda for real-time processing: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders - customers slot_name: ${PG_SLOT_NAME} stream_snapshot: true pipeline: processors: - mapping: | meta topic = meta("table") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } key: ${! json("id") } batching: count: 100 period: 1s ``` This pattern: - Uses the table name as the Redpanda topic - Batches messages for efficient delivery - Sets the message key to the row’s primary key field ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders slot_name: ${PG_SLOT_NAME} stream_snapshot: true pipeline: processors: - mapping: | root.operation = meta("operation") root.table = meta("table") root.data = this root.timestamp = now() output: aws_s3: bucket: ${S3_BUCKET} path: >- cdc/${! meta("table") }/${! timestamp_unix().format_timestamp("2006/01/02/15") }/${! uuid_v4() }.ndjson batching: count: 1000 period: 5m processors: - archive: format: lines ``` This pattern: - Organizes files by table and time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders slot_name: ${PG_SLOT_NAME} stream_snapshot: true output: switch: cases: - check: meta("operation") == "insert" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: postgres.orders.inserts - output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: postgres.orders.changes ``` This pattern: - Routes `insert` events to one Redpanda topic and all other change events to another - Supports specialized downstream consumers per operation type ## [](#configure-replication-mode)Configure replication mode The `postgres_cdc` input supports two replication modes controlled by the `stream_snapshot` field: - `stream_snapshot: true`: Captures a full snapshot of existing table data before streaming live changes. Use this when you need a complete initial load. - `stream_snapshot: false`: Skips the snapshot and streams only changes from the current WAL position. Use this when you only need new changes going forward. ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders slot_name: ${PG_SLOT_NAME} stream_snapshot: true snapshot_batch_size: 1000 (1) ``` | 1 | Number of rows to read per batch during snapshot processing. | | --- | --- | To load or reload existing table data without delaying live changes, use an incremental snapshot instead. See [Backfill tables with incremental snapshots](#backfill-tables-with-incremental-snapshots). ## [](#backfill-tables-with-incremental-snapshots)Backfill tables with incremental snapshots An incremental snapshot reads a table’s existing rows in chunks while the `postgres_cdc` input keeps streaming live changes. You request a snapshot by inserting a row into a signal table, so you can backfill or re-snapshot a table at any time without restarting the pipeline, dropping the replication slot, or changing the configuration. Incremental snapshots require Redpanda Connect 4.112.0 or later. ### [](#requirements)Requirements Before you enable incremental snapshots, check that your configuration and tables meet these requirements. The input fails to start if any of the configuration requirements are not met. - Set `stream_snapshot` to `false`. Incremental snapshots and `stream_snapshot` read the same rows, so enabling both would deliver every row twice. - Set `signal_table_name` to the name of a signal table. You request every incremental snapshot through this table. - Set `incremental_snapshot.checkpoint_cache` to a [cache resource](https://docs.redpanda.com/connect/components/caches/about/). The input stores snapshot progress in this cache, so a restart resumes the snapshot instead of starting over. Use a cache that persists across restarts, such as `redis`. With a `memory` cache, a restart discards an in-progress snapshot: the input does not resume or restart it, so you must insert a new signal. - Keep `heartbeat_interval` set to a non-zero value. The snapshot advances only when the input receives a transaction from the replication stream, and on a quiet table, heartbeats are the only transactions. Each table that you snapshot must also meet these requirements. The input rejects and logs a signal for a table that does not meet them, and continues streaming live changes. - The table must appear in `tables`, or `tables` must be empty. - The table must have a primary key. The input supports composite keys. The following primary key types are not supported: `bytea`, `interval`, `bit`, `bit varying`, and any range or multirange type. - If the table is partitioned, its publication must set `publish_via_partition_root = true`. The input creates its publication as `pglog_stream_` without this option, so set it after the input has started for the first time. For example, if `slot_name` is `my_connect_slot`, run `ALTER PUBLICATION pglog_stream_my_connect_slot SET (publish_via_partition_root = true);`. ### [](#create-the-signal-table)Create the signal table Create the signal table in the schema that you set in the `schema` field. The SQL statements on this page use `public`. If your `schema` is different, replace `public` in each statement: ```sql CREATE TABLE public.rpcn_signal_table ( id SERIAL PRIMARY KEY, type VARCHAR(32), data TEXT ); ``` The table must have the `id`, `type`, and `data` columns. At startup, the input checks that these columns exist, but not their types, so a wrong type (for example, `data JSONB` instead of `TEXT`) causes an error only when the input reads the first signal row. Do not add the signal table to `tables`. The input adds the signal table to its publication automatically and fails to start if the table is also listed in `tables`. ### [](#configure-the-input)Configure the input This configuration enables incremental snapshots for the `orders` and `customers` tables: ```yaml input: postgres_cdc: dsn: ${PG_DSN} schema: ${PG_SCHEMA} tables: - orders - customers slot_name: ${PG_SLOT_NAME} stream_snapshot: false (1) signal_table_name: rpcn_signal_table (2) incremental_snapshot: enabled: true chunk_size: 1024 (3) checkpoint_cache: snapshot_progress (4) pipeline: processors: (5) - mapping: | root = if @table == "rpcn_signal_table" { deleted() } else { this } output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("table") } key: ${! json("id") } cache_resources: - label: snapshot_progress redis: url: ${REDIS_URL} ``` | 1 | Required. Incremental snapshots and stream_snapshot are mutually exclusive. | | --- | --- | | 2 | The signal table, without the schema name. | | 3 | The number of rows to read in each chunk. The default is 1024. | | 4 | The label of the cache resource that stores snapshot progress. | | 5 | The input publishes signal rows downstream as regular insert messages. This mapping drops them by filtering on the table metadata field. | Set `REDIS_URL` to the address of your Redis server. For all incremental snapshot settings, see the [`incremental_snapshot`](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/#incremental_snapshot) field reference. ### [](#trigger-a-table-snapshot)Trigger a table snapshot To snapshot one or more tables, insert a `snapshot-execute` signal into the signal table. The `data` column is a JSON object with a `tables` key that lists the table names, without the schema name: ```sql INSERT INTO public.rpcn_signal_table (type, data) VALUES ('snapshot-execute', '{"tables": ["orders", "customers"]}'); ``` The input reads signals from the replication stream, so insert the signal after the input has created its replication slot. Check that the slot exists: ```sql SELECT slot_name, active FROM pg_replication_slots WHERE slot_name = 'my_connect_slot'; ``` Replace `my_connect_slot` with your `slot_name` value. A signal inserted before the slot exists is not in the replication stream, and the input never reads it. Each table joins the end of the snapshot queue. The input emits the snapshot rows as messages with the `operation` metadata field set to `read`, alongside the live `insert`, `update`, and `delete` messages. ### [](#re-snapshot-a-table)Re-snapshot a table The input records each table that it has snapshotted in the checkpoint cache. If you signal a table that is already recorded, the input skips the table and logs a message, so a repeated signal does not read the table again. To snapshot a table again, set `incremental_snapshot.checkpoint_cache_key` to a key that has not been used before, restart the pipeline, and insert a new `snapshot-execute` signal. Changing the key discards the record of which tables have been snapshotted. If several pipelines share the same cache, give each one a different `checkpoint_cache_key`. ### [](#handle-duplicate-rows)Handle duplicate rows An incremental snapshot can deliver the same row twice, once from the replication stream and once from the snapshot. For example, this happens when a row is inserted after replication starts but before the snapshot reaches its table. Write rows to your destination as idempotent upserts keyed by primary key, so a duplicate overwrites the earlier copy instead of creating a second record. ### [](#snapshot-tables-with-large-column-values)Snapshot tables with large column values If a table has large (TOASTed) column values, set `REPLICA IDENTITY FULL` on the table before you snapshot it: ```sql ALTER TABLE public.orders REPLICA IDENTITY FULL; ``` Under the default replica identity, PostgreSQL omits an unchanged TOAST value from an `UPDATE` message, and the input emits the `unchanged_toast_value` placeholder for that column. If a row is updated while its snapshot chunk is buffered, the input drops the snapshot copy as a duplicate, so the placeholder can be the only value your destination receives for that column. With `REPLICA IDENTITY FULL`, PostgreSQL sends the previous row, and the input reads the real value from it. You can revert the replica identity after the snapshot completes. Also set [`unchanged_toast_value`](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/#unchanged_toast_value) to a distinctive value instead of the `null` default, so downstream consumers can tell a placeholder apart from a column that is null. ## [](#troubleshoot-common-issues)Troubleshoot common issues Use these steps to diagnose and fix the most common problems with the `postgres_cdc` input. ### [](#no-events-received)No events received If no events arrive: 1. Verify logical replication is enabled: ```sql SHOW wal_level; ``` 2. Grant the required PostgreSQL privileges: ```sql ALTER ROLE your_user WITH REPLICATION; GRANT SELECT ON ALL TABLES IN SCHEMA public TO your_user; ``` > 📝 **NOTE** > > `ALTER ROLE …​ WITH REPLICATION` requires superuser privileges. Run this statement as a superuser, or have a database administrator run it on your behalf. 3. Check that `slot_name` is unique and not already in use: ```sql SELECT slot_name, active FROM pg_replication_slots; ``` ### [](#snapshot-fails-mid-run)Snapshot fails mid-run If the snapshot fails, the replication slot has an incomplete record of your database. To maintain data integrity, drop the replication slot and restart the pipeline: ```sql SELECT pg_drop_replication_slot('my_connect_slot'); ``` Replace `my_connect_slot` with your `slot_name` value. ### [](#duplicate-events)Duplicate events The `postgres_cdc` input provides at-least-once delivery. If the pipeline fails between WAL acknowledgments, events may be re-read on restart. To handle duplicates: - Use idempotent processing in downstream systems. - Deduplicate using the `lsn` metadata field. - Lower `checkpoint_limit` to reduce the window of possible duplicates. ## [](#next-steps)Next steps - [PostgreSQL CDC input reference](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 363: Retrieval-Augmented Generation (RAG) **URL**: https://docs.redpanda.com/connect/cookbooks/rag.md --- # Retrieval-Augmented Generation (RAG) --- title: Retrieval-Augmented Generation (RAG) latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: rag page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rag.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/rag.adoc description: How to configure Redpanda Connect to create a RAG pipeline, using PostgreSQL and PGVector. page-git-created-date: "2024-09-12" page-git-modified-date: "2024-09-12" --- This cookbook shows you how to create a Retrieval-Augmented Generation (RAG) pipeline, using PostgreSQL and [PGVector](https://github.com/pgvector/pgvector). Follow the cookbook to: - Create two Redpanda Connect data pipelines: one for creating vector embeddings with [Ollama](https://ollama.ai) and another for searching the data, using [PGVector](https://github.com/pgvector/pgvector). - Run the two pipelines in parallel using [streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/). ## [](#compute-the-embeddings)Compute the embeddings Start by defining the indexing pipeline, which takes textual data from Redpanda, computes vector embeddings for it, and then write it into a PostgreSQL table with a PGVector index on the embeddings column. ```bash rpk topic create articles echo '{ "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } }' | rpk topic produce articles -f '%v' ``` Your indexing pipeline can read from the Redpanda topic, using the [`kafka`](https://docs.redpanda.com/connect/components/inputs/kafka/) input: ```yaml input: kafka: addresses: [ "${REDPANDA_CLUSTER}" ] topics: [ articles ] consumer_group: rp_connect_articles_group tls: enabled: true sasl: mechanism: SCRAM-SHA-256 user: "${REDPANDA_USER}" password: "${REDPANDA_PASSWORD}" ``` Use [Nomic Embed](https://ollama.com/library/nomic-embed-text) to compute embeddings. Since each request only applies to a single document, you can scale this by making requests in parallel across document batches. To send a mapped request and map the response back into the original document, use the [`branch` processor](https://docs.redpanda.com/connect/components/processors/branch/) with a child [`ollama_embeddings`](https://docs.redpanda.com/connect/components/processors/ollama_embeddings/) processor. ```yaml pipeline: threads: -1 processors: - branch: request_map: 'root = "search_document: %s\n%s".format(this.article.title, this.article.content)' processors: - ollama_embeddings: model: nomic-embed-text result_map: 'root.article.embeddings = this' ``` With this pipeline, your processed documents should look something like this: ```yaml { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking.", "embeddings": [0.754, 0.19283, 0.231, 0.834], # This vector will actually have 768 dimensions } } ``` Now, try sending this transformed data to PostgreSQL using the [`sql_insert`](https://docs.redpanda.com/connect/components/outputs/sql_insert/) output. You can take advantage of the `init_statement` functionality to set up `pgvector` and a table to write the data to. ```yaml output: sql_insert: driver: postgres dsn: "${PG_CONNECTION_STRING}" init_statement: | CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS searchable_text ( id varchar(128) PRIMARY KEY, title text NOT NULL, body text NOT NULL, embeddings vector(768) NOT NULL ); CREATE INDEX IF NOT EXISTS text_hnsw_index ON searchable_text USING hnsw (embeddings vector_l2_ops); table: searchable_text columns: ["id", "title", "body", "embeddings"] args_mapping: "[this.article.id, this.article.title, this.article.content, this.article.embeddings.vector()]" ``` Save this pipeline as `indexing.yaml` and run it with `rpk connect run ./indexing.yaml` to make sure your PostgreSQL table is populated with embeddings. ## [](#generating-search-responses)Generating search responses To generate responses to questions with the dataset you’re creating embeddings for, use a [HTTP server input](https://docs.redpanda.com/connect/components/inputs/http_server/) to receive the questions. Since you’re running this pipeline in streams mode, the HTTP server uses [endpoints prefixed with stream identifier](https://docs.redpanda.com/connect/guides/streams_mode/about/#http-endpoints), which in this case is the same as the filename: `search`. ```yaml input: http_server: path: / allowed_verbs: [ GET ] sync_response: headers: Content-Type: application/json ``` The user query uses the query parameter `q`: `[http://localhost:4195/search?q=question_here](http://localhost:4195/search?q=question_here)`. Since query parameters are exposed as metadata in the `http_server` input, you can reference that in bloblang with `@q`. ```yaml pipeline: processors: - label: compute_embeddings ollama_embeddings: model: nomic-embed-text text: "search_query: %s".format(@q) ``` The payload is the embeddings vector. To fetch the top three most similar documents to your embeddings vector, you can write a PostgreSQL query in the [`sql_raw`](https://docs.redpanda.com/connect/components/processors/sql_raw/) processor. ```yaml - sql_raw: driver: "postgres" dsn: "${PG_CONNECTION_STRING}" query: SELECT title, body FROM searchable_text ORDER BY embeddings <-> $1 LIMIT 3 args_mapping: "[ this.vector() ]" ``` With your looked up information, as well as your initial query, you can use the [`ollama_chat` processor](https://docs.redpanda.com/connect/components/processors/ollama_chat/) to respond to the user’s question as text. ```yaml - label: generate_response ollama_chat: model: llama3.1 prompt: | Your task is to respond to user queries using the provided information. The user asked: ${! @q } Context: ${!this.map_each(row -> "%s\n%s".format(row.title, row.body)).join("\n\n")} Response: ``` Now that you’ve generated a response, you can send that back to the HTTP server as a response using [`sync_response`](https://docs.redpanda.com/connect/components/processors/sync_response/). Then, delete the message using a [bloblang mapping](https://docs.redpanda.com/connect/components/processors/mapping/) so that nothing goes to the output. ```yaml - mapping: 'root.response = content().string()' - sync_response: {} - mapping: 'root = deleted()' ``` Both pipelines are ready. Try running both of them using streams mode: `rpk connect streams indexing.yaml search.yaml`. When some documents have been indexed, you can query the system using: ```bash curl -G 'localhost:4195/search' --data-urlencode 'q=what is happening to the dogs?' | jq ``` The output should look something like: ```json { "response": "Everyone in the world woke up today shocked as their beloved pooches were silent - unable to bark." } ``` --- # Page 364: Redpanda Migrator **URL**: https://docs.redpanda.com/connect/cookbooks/redpanda_migrator.md --- # Redpanda Migrator --- title: Redpanda Migrator latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: redpanda_migrator page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/redpanda_migrator.adoc description: Move your workloads from any Kafka system to Redpanda using a single command. Redpanda Migrator lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. It's Redpanda's alternative to Kafka MirrorMaker 2. page-git-created-date: "2024-10-02" page-git-modified-date: "2026-10-15" --- With Redpanda Migrator, you can move your workloads from any Apache Kafka system to Redpanda using a single command. It lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. Redpanda Migrator is Redpanda’s alternative to Kafka MirrorMaker 2 for continuous replication into Redpanda, including timestamp-based consumer group offset translation. Redpanda Connect’s Redpanda Migrator uses the unified migrator components (available in Redpanda Connect 4.67.5+): - [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) connects to the source Kafka cluster and Schema Registry. - [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) handles all migration logic including topic creation, schema synchronization, and consumer group offset translation. > 📝 **NOTE** > > If you’re currently using the legacy `redpanda_migrator_bundle` components, see [Migrate to the Unified Redpanda Migrator](https://docs.redpanda.com/connect/guides/migration/migrate-unified-redpanda-migrator/) for migration instructions. ## [](#create-the-docker-containers)Create the Docker containers First, you need two clusters. To keep it simple, you can run the [Bitnami Kafka](https://hub.docker.com/r/bitnami/kafka) and [Schema Registry](https://hub.docker.com/r/bitnami/schema-registry) Docker containers for the source cluster and a [Redpanda Docker container](https://hub.docker.com/r/redpandadata/redpanda) for the destination cluster via [Docker Compose](https://docs.docker.com/compose). `docker-compose.yml` ```yaml services: source: image: bitnami/kafka environment: KAFKA_CFG_NODE_ID: 0 KAFKA_CFG_PROCESS_ROLES: controller,broker KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@localhost:9093 KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT KAFKA_CFG_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_DOCKER://0.0.0.0:19092,CONTROLLER://:9093 KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_DOCKER://source:19092 KAFKA_CFG_AUTHORIZER_CLASS_NAME: "org.apache.kafka.metadata.authorizer.StandardAuthorizer" KAFKA_CFG_SUPER_USERS: User:redpanda;User:ANONYMOUS ports: - 9092:9092 - 19092:19092 healthcheck: test: [ "CMD", "kafka-topics.sh", "--bootstrap-server=localhost:9092", "--list" ] start_period: 5s interval: 3s init_source: image: bitnami/kafka working_dir: /opt/bitnami/kafka/bin entrypoint: /bin/bash depends_on: source: condition: service_healthy command: > -c "kafka-topics.sh --bootstrap-server source:19092 --create --if-not-exists --topic foo --replication-factor=1 --partitions=2 && kafka-topics.sh --bootstrap-server source:19092 --create --if-not-exists --topic bar --replication-factor=1 --partitions=2 && echo 'Created topics:' && kafka-topics.sh --bootstrap-server source:19092 --list --exclude-internal && kafka-acls.sh --bootstrap-server source:19092 --add --allow-principal User:redpanda --operation Read --topic foo && kafka-acls.sh --bootstrap-server source:19092 --add --deny-principal User:redpanda --operation Read --topic bar echo 'Created ACLs:' && kafka-acls.sh --bootstrap-server source:19092 --list" source_schema_registry: image: bitnami/schema-registry environment: SCHEMA_REGISTRY_KAFKA_BROKERS: PLAINTEXT://source:19092 ports: - 8081:8081 depends_on: source: condition: service_healthy destination: image: redpandadata/redpanda command: - redpanda - start - --node-id 0 - --mode dev-container - --set rpk.additional_start_flags=[--reactor-backend=epoll] - --set redpanda.auto_create_topics_enabled=false - --kafka-addr 0.0.0.0:9093 - --advertise-kafka-addr localhost:9093 - --schema-registry-addr 0.0.0.0:8081 ports: - 8082:8081 - 9093:9093 - 9645:9644 ``` ```bash docker compose -f docker-compose.yml up --force-recreate -V ``` > 📝 **NOTE** > > This command uses an `init` container to create two topics, `foo` and `bar`, each with two partitions and an associated ACL. ## [](#create-schemas)Create schemas When the demo clusters are up and running, use [curl](https://curl.se) to create a schema for each topic in the `source` cluster. ```bash curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"name\": \"Foo\", \"type\": \"record\", \"fields\": [{\"name\": \"data\", \"type\": \"int\"}]}"}' http://localhost:8081/subjects/foo/versions curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"name\": \"Bar\", \"type\": \"record\", \"fields\": [{\"name\": \"data\", \"type\": \"int\"}]}"}' http://localhost:8081/subjects/bar/versions ``` ## [](#generate-messages)Generate messages Let’s simulate an application with a producer and consumer actively publishing and reading messages on the `source` cluster. You can use Redpanda Connect to generate some Avro-encoded messages and push them to the two topics from the `source` cluster. `generate_data.yaml` ```yaml http: enabled: false input: sequence: inputs: - generate: mapping: | let msg = counter() root.data = $msg meta kafka_topic = match $msg % 2 { 0 => "foo" 1 => "bar" } interval: 1s count: 0 batch_size: 1 processors: - schema_registry_encode: url: "http://localhost:8081" subject: ${! metadata("kafka_topic") } avro_raw_json: true output: kafka_franz: seed_brokers: [ "localhost:9092" ] topic: ${! @kafka_topic } partitioner: manual partition: ${! random_int(min:0, max:1) } ``` Now, run this command to start the pipeline, and leave it running: ```bash rpk connect run generate_data.yaml ``` Next, add a Redpanda Connect consumer, which reads messages from the `source` cluster topics, and leave it running. This consumer uses the `foobar` consumer group, which is reused in a later step when consuming from the `destination` cluster. `read_data_source.yaml` ```yaml http: enabled: false input: kafka_franz: seed_brokers: [ "localhost:9092" ] topics: - '^[^_]' # Skip topics which start with `_` regexp_topics: true consumer_group: foobar processors: - schema_registry_decode: url: "http://localhost:8081" avro_raw_json: true output: stdout: {} processors: - mapping: | root = this.merge({"count": counter(), "topic": @kafka_topic, "partition": @kafka_partition}) ``` Launch the `source` consumer pipeline, and leave it running: ```bash rpk connect run read_data_source.yaml ``` At this point, the `source` cluster has some data in both `foo` and `bar` topics, and the consumer prints the messages it reads from these topics to `stdout`. ## [](#required-permissions)Required permissions This cookbook authenticates as a superuser for simplicity. In production, when the source and destination clusters enforce authorization (ACLs), a basic consumer or producer ACL is not sufficient for Redpanda Migrator. The migrator authenticates to the source cluster with the credentials in the `redpanda_migrator` input, and to the destination cluster with the credentials in the `redpanda_migrator` output, so grant ACLs to each principal as follows. > ❗ **IMPORTANT** > > To recreate each topic on the destination with matching settings, Redpanda Migrator reads the source topic’s configuration with a `DescribeConfigs` request, which requires the `DESCRIBE_CONFIGS` operation on the topic. > > A consumer ACL (`READ`) implicitly grants `DESCRIBE`, but it does **not** grant `DESCRIBE_CONFIGS`. If the source principal has only `READ`, the migrator consumes messages successfully but fails to create topics, logging an error such as: > > ```text > level=error msg="Failed to send message to redpanda_migrator: creating records: sync topics: > create topic : get topic details : TOPIC_AUTHORIZATION_FAILED: > Not authorized to access topics: [Topic authorization failed.]" > ``` > > Despite the `create topic` wording, this failure is the `DescribeConfigs` read against the **source** cluster, not the topic creation on the destination. ### [](#source-cluster)Source cluster Grant the source principal (the `redpanda_migrator` input credentials) these ACLs: | Resource | Operations | Purpose | | --- | --- | --- | | Topic (migrated topics) | READ, DESCRIBE_CONFIGS | Consume records (READ, which also grants DESCRIBE for metadata and offsets) and read topic configurations to replicate them (DESCRIBE_CONFIGS). | | Group (the input’s consumer_group) | READ | Join the migrator’s own consumer group and track progress. | | Group (migrated groups) | DESCRIBE | Read source consumer group offsets. Required when consumer group migration is enabled (consumer_groups.enabled, the default). | | Cluster | DESCRIBE | List source consumer groups. Also required to read source ACLs when sync_topic_acls is enabled on the output. | For example, to grant the least-privilege source ACLs to `User:migrator` with `rpk`: ```bash # Consume records and read topic configs (READ also grants DESCRIBE; DESCRIBE_CONFIGS does not come with READ) rpk security acl create --allow-principal User:migrator \ --operation read,describe_configs \ --topic # READ on the migrator's own consumer group rpk security acl create --allow-principal User:migrator \ --operation read \ --group # DESCRIBE on the groups being migrated (omit if consumer_groups.enabled is false) rpk security acl create --allow-principal User:migrator \ --operation describe \ --group # List consumer groups (and describe ACLs if sync_topic_acls is enabled) rpk security acl create --allow-principal User:migrator \ --operation describe \ --cluster ``` ### [](#destination-cluster)Destination cluster Grant the destination principal (the `redpanda_migrator` output credentials) these ACLs: | Resource | Operations | Purpose | | --- | --- | --- | | Topic (migrated topics) | CREATE, WRITE, ALTER, DESCRIBE_CONFIGS, READ | Create topics (CREATE and DESCRIBE_CONFIGS), produce records (WRITE), and add partitions (ALTER). These operations also grant DESCRIBE. READ translates and commits consumer group offsets. Omit it if consumer_groups.enabled is false. | | Cluster | CREATE | Allow creating destination topics whose names are not known in advance. Use instead of per-topic CREATE. | | Group (migrated groups) | READ | Commit translated consumer group offsets. Required when consumer group migration is enabled (consumer_groups.enabled, the default). | | Cluster | ALTER | Create migrated ACLs on the destination. Required only when sync_topic_acls is enabled. | > 💡 **TIP** > > Run `rpk security acl --help-operations` to see which ACL operation each Kafka request requires. ## [](#configure-and-start-redpanda-migrator)Configure and start Redpanda Migrator The unified Redpanda Migrator does the following: - The `redpanda_migrator` input connects to the source Kafka cluster and Schema Registry to consume messages and schema information. - The `redpanda_migrator` output handles all migration logic: - Schema migration: reads schemas from the source Schema Registry and synchronizes them to the destination. - Topic creation: automatically creates destination topics that don’t exist with proper configurations. - ACL migration: migrates access control lists according to the migration rules. - Message streaming: processes and routes messages from source to destination topics. - Consumer group offset translation: maps source consumer group offsets to equivalent destination positions. - If new topics are created in the source cluster while the migrator is running, they are migrated when messages are written to them. ACL migration for topics adheres to the following principles: - `ALLOW WRITE` ACLs for topics are not migrated - `ALLOW ALL` ACLs for topics are downgraded to `ALLOW READ` - Group ACLs are not migrated > 📝 **NOTE** > > Changing topic configurations, such as partition count, isn’t currently supported. Now, use the following unified Redpanda Migrator configuration. See the [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) and [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) docs for details. `redpanda_migrator.yaml` ```yaml input: label: "migration_pipeline" (1) redpanda_migrator: # Source Kafka settings seed_brokers: [ "localhost:9092" ] topics: - '^[^_]' # Skip internal topics which start with `_` regexp_topics: true consumer_group: migrator # Source Schema Registry settings schema_registry: url: http://localhost:8081 output: label: "migration_pipeline" (2) redpanda_migrator: # Destination Redpanda settings seed_brokers: [ "localhost:9093" ] # Destination Schema Registry and migration settings schema_registry: url: http://localhost:8082 include_deleted: true translate_ids: true # Consumer group migration settings consumer_groups: enabled: true interval: 30s metrics: prometheus: {} mapping: | meta label = if this == "input_redpanda_migrator_lag" { "source" } ``` > 💡 **TIP** > > Label names must be between 3 and 128 characters and can only contain alphanumeric characters, hyphens, and underscores (`A-Za-z0-9-_`). Launch the unified Redpanda Migrator pipeline, and leave it running: ```bash rpk connect run redpanda_migrator.yaml ``` ## [](#check-the-status-of-migrated-topics)Check the status of migrated topics You can use the Redpanda [`rpk` CLI tool](https://docs.redpanda.com/streaming/current/get-started/rpk/) to check which topics and ACLs have been migrated to the `destination` cluster. You can quickly [install `rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) if you don’t already have it. > 📝 **NOTE** > > For now, users require manual migration. However, this step is not required for the current demo. Similarly, roles are specific to Redpanda and, for now, also require manual migration if the `source` cluster is based on Redpanda. ```bash rpk -X brokers=localhost:9093 topic list NAME PARTITIONS REPLICAS _schemas 1 1 bar 2 1 foo 2 1 rpk -X brokers=localhost:9093 security acl list PRINCIPAL HOST RESOURCE-TYPE RESOURCE-NAME RESOURCE-PATTERN-TYPE OPERATION PERMISSION ERROR User:redpanda * TOPIC bar LITERAL READ DENY User:redpanda * TOPIC foo LITERAL READ ALLOW ``` ## [](#check-metrics-to-monitor-progress)Check metrics to monitor progress Redpanda Connect provides a comprehensive suite of metrics in various formats, such as Prometheus, which you can use to monitor its performance in your observability stack. Besides the [standard Redpanda Connect metrics](https://docs.redpanda.com/connect/components/metrics/about/#metric-names), the `redpanda_migrator` input also emits an `input_redpanda_migrator_lag` metric for monitoring the migration progress of each topic and partition. ```bash curl http://localhost:4195/metrics ... # HELP input_redpanda_migrator_lag Benthos Gauge metric # TYPE input_redpanda_migrator_lag gauge input_redpanda_migrator_lag{label="source",partition="0",path="root.input.sequence.broker.inputs.0",topic="__consumer_offsets"} 0 input_redpanda_migrator_lag{label="source",partition="0",path="root.input.sequence.broker.inputs.0",topic="bar"} 0 input_redpanda_migrator_lag{label="source",partition="0",path="root.input.sequence.broker.inputs.0",topic="foo"} 0 input_redpanda_migrator_lag{label="source",partition="1",path="root.input.sequence.broker.inputs.0",topic="__consumer_offsets"} 0 input_redpanda_migrator_lag{label="source",partition="1",path="root.input.sequence.broker.inputs.0",topic="bar"} 1 input_redpanda_migrator_lag{label="source",partition="1",path="root.input.sequence.broker.inputs.0",topic="foo"} 0 ... ``` ## [](#read-from-the-migrated-topics)Read from the migrated topics Stop the `read_data_source.yaml` consumer you started earlier and, afterwards, start a similar consumer for the `destination` cluster. Before starting the consumer up on the `destination` cluster, make sure you give the migrator bundle some time to replicate the translated offset. `read_data_destination.yaml` ```yaml http: enabled: false input: kafka_franz: seed_brokers: [ "localhost:9093" ] topics: - '^[^_]' # Skip topics which start with `_` regexp_topics: true consumer_group: foobar processors: - schema_registry_decode: url: "http://localhost:8082" avro_raw_json: true output: stdout: {} processors: - mapping: | root = this.merge({"count": counter(), "topic": @kafka_topic, "partition": @kafka_partition}) ``` Now launch the `destination` consumer pipeline, and leave it running: ```bash rpk connect run read_data_destination.yaml ``` The `source` cluster consumer uses the same `foobar` consumer group. This consumer resumes reading messages from where the `source` consumer left off. Redpanda Migrator performs offset remapping when migrating consumer group offsets to the `destination` cluster. While more sophisticated approaches are possible, Redpanda chose to use a simple timestamp-based approach. So, for each migrated offset, the `destination` cluster is queried to find the latest offset before the received offset timestamp. Redpanda Migrator then writes this offset as the `destination` consumer group offset for the corresponding topic and partition pair. Although the timestamp-based approach doesn’t guarantee exactly-once delivery, it minimizes the likelihood of message duplication and avoids the need for complex and error-prone offset remapping logic. --- # Page 365: Ingest data into Snowflake **URL**: https://docs.redpanda.com/connect/cookbooks/snowflake_ingestion.md --- # Ingest data into Snowflake --- title: Ingest data into Snowflake latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: snowflake_ingestion page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: snowflake_ingestion.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/snowflake_ingestion.adoc description: Configure Redpanda Connect to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. page-git-created-date: "2025-01-28" page-git-modified-date: "2026-05-26" --- Configure a Redpanda Connect pipeline to generate and write data into a local Redpanda topic, and then ingest that data into [Snowflake](https://www.snowflake.com/en/) using [Snowpipe Streaming](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview). See also: [Ingest data into Snowflake](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/snowflake_ingestion/) using Redpanda Cloud ## [](#prerequisites)Prerequisites - [`rpk` installed](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) - A [Snowflake account](https://trial.snowflake.com/) - `openssl` command-line tool ## [](#set-up-your-redpanda-cluster)Set up your Redpanda cluster Run [rpk container start](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container-start/) to create a local Redpanda cluster for development: ```none rpk container start ``` Next, create a `demo_topic` to use as the data source for ingesting data into Snowflake: ```bash rpk topic create demo_topic ``` Create a user with minimal [ACLs](https://docs.redpanda.com/streaming/current/manage/security/authorization/acl/) to run the ingestion pipeline into Snowflake: ```bash rpk security user create ingestion_user --password Testing1234 ``` Now that the user exists, give them read permissions to `demo_topic`, as well as full control over any consumer group with the prefix `redpanda_connect`: ```bash rpk security acl create --allow-principal ingestion_user --operation read --topic demo_topic rpk security acl create --allow-principal ingestion_user --resource-pattern-type prefixed --operation all --group redpanda_connect ``` ## [](#set-up-your-snowflake-account)Set up your Snowflake account Log in to your Snowflake account with a user who has the ACCOUNTADMIN role. Then, run the following SQL commands in a worksheet. They set up another user with minimal permissions to write data into a specified database and schema, ready for streaming data to Snowflake. ```sql -- Set default values for multiple variables SET PWD = 'Test1234567'; SET USER = 'STREAMING_USER'; SET DB = 'STREAMING_DB'; SET ROLE = 'REDPANDA_CONNECT'; SET WH = 'STREAMING_WH'; USE ROLE ACCOUNTADMIN; -- Create users CREATE USER IF NOT EXISTS IDENTIFIER($USER) PASSWORD=$PWD COMMENT='STREAMING USER FOR REDPANDA CONNECT'; -- Create roles CREATE OR REPLACE ROLE IDENTIFIER($ROLE); -- Create the destination database and virtual warehouse CREATE DATABASE IF NOT EXISTS IDENTIFIER($DB); USE IDENTIFIER($DB); CREATE OR REPLACE WAREHOUSE IDENTIFIER($WH) WITH WAREHOUSE_SIZE = 'SMALL'; -- Grant privileges GRANT CREATE WAREHOUSE ON ACCOUNT TO ROLE IDENTIFIER($ROLE); GRANT ROLE IDENTIFIER($ROLE) TO USER IDENTIFIER($USER); GRANT OWNERSHIP ON DATABASE IDENTIFIER($DB) TO ROLE IDENTIFIER($ROLE); GRANT USAGE ON WAREHOUSE IDENTIFIER($WH) TO ROLE IDENTIFIER($ROLE); -- Set defaults ALTER USER IDENTIFIER($USER) SET DEFAULT_ROLE=$ROLE; ALTER USER IDENTIFIER($USER) SET DEFAULT_WAREHOUSE=$WH; -- Run the following commands to find your account identifier. Copy it down for later use. -- It will be something like `organization_name-account_name` -- e.g. ykmxgak-wyb52636 WITH HOSTLIST AS (SELECT * FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$allowlist())))) SELECT REPLACE(VALUE:host,'.snowflakecomputing.com','') AS ACCOUNT_IDENTIFIER FROM HOSTLIST WHERE VALUE:type = 'SNOWFLAKE_DEPLOYMENT_REGIONLESS'; ``` ### [](#create-an-rsa-key-pair)Create an RSA key pair Create an [RSA key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth) using `openssl` to authenticate Redpanda Connect to Snowflake. When you’re prompted to give an encryption password, record it for later. ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -passout pass:Testing123 -out rsa_key.p8 ``` Create a public key. You’re prompted to enter your encryption password. ```bash openssl rsa -in rsa_key.p8 -pubout -passout pass:Testing123 -out rsa_key.pub ``` To register the public key in Snowflake, remove the public key delimiters and output only the base64-encoded portion of the PEM file. Run the following bash command to print it: ```bash cat rsa_key.pub | sed -e '1d' -e '$d' | tr -d '\n' ``` In the Snowflake worksheet, add the output of the bash command you just ran to the following SQL command and execute it: ```sql use role accountadmin; alter user streaming_user set rsa_public_key='< PubKeyWithoutDelimiters >'; ``` ### [](#create-a-schema-using-streaming_user)Create a schema using `streaming_user` Log out of Snowflake and sign back in as the default user (`streaming_user`) with the associated password (default: `Test1234567`). You created these credentials in [Set up your Snowflake account](#set-up-your-snowflake-account). Run the following SQL commands in a worksheet to create a schema (e.g. `STREAMING_SCHEMA`) in the default database (e.g. `STREAMING_DB`): ```sql SET DB = 'STREAMING_DB'; SET SCHEMA = 'STREAMING_SCHEMA'; USE IDENTIFIER($DB); CREATE OR REPLACE SCHEMA IDENTIFIER($SCHEMA); ``` ## [](#create-a-pipeline-from-your-redpanda-cluster-to-snowflake)Create a pipeline from your Redpanda cluster to Snowflake You can now create the pipeline locally and inject [secrets](https://docs.redpanda.com/connect/configuration/secrets/) using environment variables. First, create a `connect.yaml` file: ```yaml input: # Read data from the local `demo_topic` kafka_franz: seed_brokers: ["localhost:9092"] topics: ["demo_topic"] consumer_group: "redpanda_connect_to_snowflake" tls: {enabled: true} checkpoint_limit: 4096 sasl: - mechanism: SCRAM-SHA-256 username: ingestion_user password: ${REDPANDA_PASS} # Define a batching policy. For this cookbook, create small batches, # but in a production environment use the largest file size you can. batching: count: 100 # Collect 10 messages before flushing period: 10s # or after 10 seconds, which ever comes first output: snowflake_streaming: # Replace this placeholder with your account identifier account: "< OrgName-AccountName >" user: STREAMING_USER role: REDPANDA_CONNECT database: STREAMING_DB schema: STREAMING_SCHEMA table: STREAMING_DATA # Inject your private key and password private_key_file: rsa_key.p8 private_key_pass: "${SNOWFLAKE_KEY_PASS}" schema_evolution: enabled: true max_in_flight: 1 ``` Now run the pipeline, and any JSON data produced into the topic is streamed into Snowflake with minimal latency. ```bash REDPANDA_PASS=Testing1234 SNOWFLAKE_KEY_PASS=Testing123 rpk connect run ./connect.yaml ``` You now can produce some data using `rpk` to test that everything works: ```bash echo '{"animal":"redpanda","attributes":"cute","age":6}' | rpk topic produce demo_topic -f '%v\n' echo '{"animal":"polar bear","attributes":"cool","age":13}' | rpk topic produce demo_topic -f '%v\n' echo '{"animal":"unicorn","attributes":"rare","age":999}' | rpk topic produce demo_topic -f '%v\n' ``` The data produced into the `demo_topic` is consumed and streamed into Snowflake in seconds. Go back to the Snowflake worksheet and run the following query to see data arrive in Snowflake with the schema from the JSON data you produced. ```sql SELECT * FROM STREAMING_DB.STREAMING_SCHEMA.STREAMING_DATA LIMIT 50; ``` See also: - The [`kafka_franz` input](https://docs.redpanda.com/connect/components/inputs/kafka_franz/) - The [`snowflake_streaming`](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) output --- # Page 366: TigerBeetle CDC Patterns **URL**: https://docs.redpanda.com/connect/cookbooks/tigerbeetle_cdc.md --- # TigerBeetle CDC Patterns --- title: TigerBeetle CDC Patterns latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: tigerbeetle_cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tigerbeetle_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/tigerbeetle_cdc.adoc description: Learn how to capture, filter, transform, and route TigerBeetle change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing TigerBeetle CDC events learning-objective-2: Adapt integration patterns to route financial ledger events to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming financial transfer events page-git-created-date: "2026-06-17" page-git-modified-date: "2026-10-14" --- The `tigerbeetle_cdc` input streams financial ledger events from a [TigerBeetle](https://tigerbeetle.com/) cluster in real time. Use these patterns to filter, transform, and route transfer events to Redpanda, S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing TigerBeetle CDC events - Adapt integration patterns to route financial ledger events to Redpanda and S3 - Identify patterns for filtering and transforming financial transfer events > 📝 **NOTE** > > The `tigerbeetle_cdc` input is in beta. The API is subject to change. ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#cgo-enabled-binary)Cgo-enabled binary The `tigerbeetle_cdc` input requires a cgo-enabled build of Redpanda Connect. The `rpk` CLI and the standard Docker image do not include the `tigerbeetle_cdc` input. To get a cgo-enabled binary: - [Download a prebuilt cgo-enabled binary](https://docs.redpanda.com/connect/install/prebuilt-binary/), or - [Build Redpanda Connect from source](https://docs.redpanda.com/connect/install/build-from-source/) with cgo enabled. ### [](#tigerbeetle-cluster)TigerBeetle cluster The `tigerbeetle_cdc` input requires a TigerBeetle cluster running version 0.16.57 or later. The Redpanda Connect TigerBeetle client version must not be newer than the cluster version. For a local cluster, see the [TigerBeetle documentation](https://docs.tigerbeetle.com/). ### [](#progress-cache)Progress cache The `tigerbeetle_cdc` input requires a [cache](https://docs.redpanda.com/connect/components/caches/about/) to store the last acknowledged event timestamp between restarts. The examples in this cookbook use a Redis cache: ```bash export REDIS_URL=redis://localhost:6379 ``` For production use, any persistent cache supported by Redpanda Connect works, such as [`aws_dynamodb`](https://docs.redpanda.com/connect/components/caches/aws_dynamodb/) or [`sql`](https://docs.redpanda.com/connect/components/caches/sql/). ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for configuration: ```bash export TB_CLUSTER_ID="1" (1) export TB_REPLICA_1="192.168.1.10:3000" (2) export TB_REPLICA_2="192.168.1.11:3000" export TB_REPLICA_3="192.168.1.12:3000" export REDIS_URL=redis://localhost:6379 (3) export REDPANDA_BROKERS=localhost:9092 (4) export S3_BUCKET=cdc-archive (5) ``` | 1 | The unique 128-bit TigerBeetle cluster ID. Small integers are valid (for example, 1 represents the 128-bit value 1). | | --- | --- | | 2 | The addresses of each replica in host:port format. A standard cluster has three replicas. | | 3 | The Redis URL for progress tracking. | | 4 | The Redpanda broker addresses (for Redpanda output examples). | | 5 | The Amazon S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all events from a TigerBeetle cluster and outputs them with metadata: ```yaml input: tigerbeetle_cdc: cluster_id: ${TB_CLUSTER_ID} addresses: - ${TB_REPLICA_1} - ${TB_REPLICA_2} - ${TB_REPLICA_3} progress_cache: redis_cache pipeline: processors: - mapping: | root.event_type = meta("event_type") root.ledger = meta("ledger") root.transfer_code = meta("transfer_code") root.timestamp_ms = meta("timestamp_ms") root.transfer = this.transfer root.debit_account = this.debit_account root.credit_account = this.credit_account output: stdout: codec: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` For details on the event structure and available metadata fields, see the [metadata](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/#metadata) section in the connector reference. Each event contains a full snapshot of the transfer and both the debit and credit accounts at the time of the event. ## [](#filter-cdc-events)Filter CDC events Filter events to process only settled transfers (`single_phase` for immediate and `two_phase_posted` for completed two-phase) using the `event_type` metadata field: ```yaml input: tigerbeetle_cdc: cluster_id: ${TB_CLUSTER_ID} addresses: - ${TB_REPLICA_1} - ${TB_REPLICA_2} - ${TB_REPLICA_3} progress_cache: redis_cache pipeline: processors: - mapping: | root = if meta("event_type") != "single_phase" && meta("event_type") != "two_phase_posted" { deleted() } - mapping: | root.event_type = meta("event_type") root.ledger = meta("ledger") root.transfer_id = this.transfer.id root.amount = this.transfer.amount root.debit_account_id = this.debit_account.id root.credit_account_id = this.credit_account.id root.timestamp_ms = meta("timestamp_ms") output: stdout: codec: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Keeps only `single_phase` and `two_phase_posted` events, dropping pending, voided, and expired events - Flattens the event to key transfer fields and a millisecond timestamp ## [](#route-to-redpanda)Route to Redpanda Stream TigerBeetle transfer events to Redpanda, partitioned by ledger code: ```yaml input: tigerbeetle_cdc: cluster_id: ${TB_CLUSTER_ID} addresses: - ${TB_REPLICA_1} - ${TB_REPLICA_2} - ${TB_REPLICA_3} progress_cache: redis_cache pipeline: processors: - mapping: | meta topic = "transfers." + meta("ledger") output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: ${! meta("topic") } key: ${! json("transfer.id") } batching: count: 100 period: 1s cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Creates a Redpanda topic per ledger (for example, `transfers.1`, `transfers.2`) - Batches messages for efficient delivery - Sets the message key to the transfer ID ## [](#route-to-s3)Route to S3 Archive transfer events to Amazon S3 for long-term storage and analytics: ```yaml input: tigerbeetle_cdc: cluster_id: ${TB_CLUSTER_ID} addresses: - ${TB_REPLICA_1} - ${TB_REPLICA_2} - ${TB_REPLICA_3} progress_cache: redis_cache pipeline: processors: - mapping: | root.event_type = meta("event_type") root.ledger = meta("ledger") root.transfer = this.transfer root.debit_account = this.debit_account root.credit_account = this.credit_account root.timestamp_ms = meta("timestamp_ms") output: aws_s3: bucket: ${S3_BUCKET} path: >- cdc/ledger/${! meta("ledger") }/${! timestamp_unix().format_timestamp("2006/01/02/15") }/${! uuid_v4() }.ndjson batching: count: 1000 period: 5m processors: - archive: format: lines cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Organizes files by ledger code and time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route single-phase and two-phase transfers to separate Redpanda topics: ```yaml input: tigerbeetle_cdc: cluster_id: ${TB_CLUSTER_ID} addresses: - ${TB_REPLICA_1} - ${TB_REPLICA_2} - ${TB_REPLICA_3} progress_cache: redis_cache output: switch: cases: - check: meta("event_type") == "single_phase" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: tigerbeetle.single_phase - check: meta("event_type") == "two_phase_posted" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: tigerbeetle.two_phase - output: drop: {} cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` This pattern: - Routes `single_phase` events (immediate settlements) to one topic - Routes `two_phase_posted` events (settled two-phase transfers) to another topic - Drops other two-phase events (`two_phase_pending`, `two_phase_voided`, `two_phase_expired`) that have not yet settled - Supports specialized consumers for each settlement model ## [](#resume-from-a-timestamp)Resume from a timestamp By default, the `tigerbeetle_cdc` input streams all events available in the cluster. To start from a specific point in time, set `timestamp_initial` to a TigerBeetle nanosecond timestamp: ```yaml input: tigerbeetle_cdc: cluster_id: ${TB_CLUSTER_ID} addresses: - ${TB_REPLICA_1} - ${TB_REPLICA_2} - ${TB_REPLICA_3} progress_cache: redis_cache timestamp_initial: "1745328372758695656" (1) cache_resources: - label: redis_cache redis: url: ${REDIS_URL} ``` | 1 | Start from events at or after this nanosecond timestamp. Ignored if progress_cache already contains a more recent acknowledged timestamp. | | --- | --- | ## [](#troubleshoot-common-issues)Troubleshoot common issues Use these steps to diagnose and fix common problems with the `tigerbeetle_cdc` input. ### [](#component-not-available)Component not available If the pipeline fails with an error that `tigerbeetle_cdc` is not a recognized input: - Confirm you are using a cgo-enabled build of Redpanda Connect, not `rpk` or the standard Docker image. - Verify the binary version supports `tigerbeetle_cdc` (introduced in version 4.65.0). ### [](#cannot-connect-to-cluster)Cannot connect to cluster If the pipeline fails to connect to TigerBeetle: - Verify the `addresses` list contains the correct IP and port for each replica. - Confirm `cluster_id` matches the ID used when creating the cluster. - Check network access between Redpanda Connect and the TigerBeetle replicas. ### [](#duplicate-events-after-restart)Duplicate events after restart The `tigerbeetle_cdc` input provides at-least-once delivery. During crash recovery, the pipeline may replay events that were already delivered. To handle duplicates: - Use idempotent processing in downstream systems. - Deduplicate using the `timestamp` metadata field, which is unique per event with nanosecond precision. ## [](#next-steps)Next steps - [TigerBeetle CDC input reference](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/) - [Redis cache](https://docs.redpanda.com/connect/components/caches/redis/) - [Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 367: Get Started **URL**: https://docs.redpanda.com/connect/get-started.md --- # Get Started --- title: Get Started latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/index.adoc description: Explore these topics to get started with Redpanda Connect. page-git-created-date: "2024-11-14" page-git-modified-date: "2024-11-14" --- Explore these topics to get started with Redpanda Connect. - [Introduction to Redpanda Connect](about/) Learn about Redpanda Connect, a declarative connector framework for building data pipelines with simple, chained, stateless processing steps. - [What’s New in Redpanda Connect](whats-new/) Summary of new features in Redpanda Connect. - [Quickstarts](quickstarts/) Use your preferred tool to spin up a local instance of Redpanda Connect and get hands-on experience building your first pipelines. - [Enterprise Licensing](licensing/) Learn how to purchase and apply an Enterprise Edition license. - [Redpanda Connect Versioning](versioning/) Learn how Redpanda Connect implements versioning. --- # Page 368: Introduction to Redpanda Connect **URL**: https://docs.redpanda.com/connect/get-started/about.md --- # Introduction to Redpanda Connect --- title: Introduction to Redpanda Connect latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/about.adoc description: Learn about Redpanda Connect, a declarative connector framework for building data pipelines with simple, chained, stateless processing steps. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-05-26" --- Redpanda Connect is a declarative connector framework that solves a wide range of data engineering problems with simple, chained, stateless [processing steps](https://docs.redpanda.com/connect/components/processors/about/). It implements transaction-based resiliency with back pressure, so when connecting to at-least-once sources and sinks it’s able to guarantee at-least-once delivery without needing to persist messages during transit. It’s [simple to deploy](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/), comes with a wide range of [connectors](https://docs.redpanda.com/connect/components/about/), and is data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative. ## [](#next-steps)Next steps - [Get started](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) - [View all connectors](https://docs.redpanda.com/connect/components/about/) --- # Page 369: Enterprise Licensing **URL**: https://docs.redpanda.com/connect/get-started/licensing.md --- # Enterprise Licensing --- title: Enterprise Licensing latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: licensing page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: licensing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/licensing.adoc description: Learn how to purchase and apply an Enterprise Edition license. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-08-17" --- You need a valid Enterprise Edition license to use the following enterprise features: | Feature | Description | Restrictions Without Valid License | | --- | --- | --- | | Allow or deny lists | Limit the Redpanda Connect components that users can run within data pipelines on a Redpanda Connect instance. | No change. | | Enterprise connectors | Additional inputs, outputs, and processors available only to enterprise customers. | All enterprise connectors are blocked. | | FIPS compliance for Redpanda Connect | Run Redpanda Connect using a FIPS-compliant version of rpk, the Redpanda command-line interface (CLI). | No change. | | Redpanda Connect configuration service | A configuration block that you can use to send logs and status events to a topic on a Redpanda cluster. | No change. | | Secrets management | Retrieve secrets values from a remote system, such as a secret management solution, without setting environment variables. | No change. | ## [](#get-an-enterprise-license)Get an enterprise license You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. ### [](#product-scoped-licenses)Product-scoped licenses In Redpanda Connect 4.78.0 and later, licenses specify which Redpanda products they cover. To unlock enterprise features, your license must include the Redpanda Connect product. Licenses issued specifically for Redpanda Connect include it, and Enterprise Edition licenses issued before product scoping was introduced are valid for all products. If you are not sure which products your license covers, contact your Redpanda account team. ## [](#apply-a-license-key-to-redpanda-connect)Apply a license key to Redpanda Connect When you have downloaded the license key, you can apply it using either the CLI (`rpk`) or Helm on Kubernetes. Redpanda Connect looks for a license in the following order and uses the first one it finds: 1. A license string passed with the `--redpanda-license` flag or set in the `REDPANDA_LICENSE` environment variable. The flag takes precedence over the environment variable. 2. The license file set in the `REDPANDA_LICENSE_FILEPATH` environment variable. 3. The default license file at `/etc/redpanda/redpanda.license`. ### rpk (local or VM) If you have a license key file, either: - Save the license key to `/etc/redpanda/redpanda.license`. - Set the environment variable `REDPANDA_LICENSE_FILEPATH` to point to the file where the license key is stored. If you have a license string, either: - Set the environment variable `REDPANDA_LICENSE` to the value of the full license string. - Run the following command using the full license string: ```bash rpk connect run --redpanda-license ./.yaml ``` Replace the following placeholders: - ``: The full license key string. - ``: The name of your Redpanda Connect configuration file. ### Helm (Kubernetes) To enable enterprise features, you must provide a license key as a Kubernetes Secret and reference it in your deployment configuration. 1. Create the Secret: ```bash kubectl create secret generic redpanda-connect-license \ --from-literal=REDPANDA_LICENSE= \ --namespace ``` 2. Reference the license secret in your `values.yaml` or as a Helm value: ```yaml envFrom: - secretRef: name: redpanda-connect-license ``` This configuration mounts the license key as the `REDPANDA_LICENSE` environment variable, allowing Redpanda Connect to use enterprise features. ## [](#verify-a-license)Verify a license Redpanda Connect checks the license key status at runtime. If the license key is unavailable or has expired, you are blocked from using enterprise connectors. You can view the license key’s expiration date at any time in the Redpanda Connect logs: ```none INFO Running main config from specified file @service=benthos benthos_version=4.50.0 path=connect_2.yaml INFO Successfully loaded Redpanda license @service=benthos expires_at="2025-04-18T15:41:56+01:00" license_org=67XXX license_type="enterprise" ``` If no valid enterprise license is loaded, or your license does not cover Redpanda Connect, enterprise components fail with the following error: this feature requires a valid Redpanda Enterprise Edition license that includes the Connect product. For more information check out: https://docs.redpanda.com/redpanda-connect/get-started/licensing/ If no license is loaded at all, see [Apply a license key to Redpanda Connect](#apply-a-license-key-to-redpanda-connect) earlier on this page. If your license is valid but doesn’t cover Redpanda Connect, contact your Redpanda account team to add the Redpanda Connect product to your license. When a valid enterprise license is loaded, Redpanda Connect also exports the `redpanda_cluster_features_enterprise_license_expiry_sec` metric, which reports the number of seconds until the license expires. You can alert on this metric to catch expiring licenses before enterprise connectors stop working. ## [](#next-steps)Next steps - [Get started with Redpanda Connect](https://docs.redpanda.com/connect/get-started/quickstarts/). - [Explore enterprise connectors](https://docs.redpanda.com/connect/components/about/) to enhance your data pipelines. --- # Page 370: Quickstarts **URL**: https://docs.redpanda.com/connect/get-started/quickstarts.md --- # Quickstarts --- title: Quickstarts latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: quickstarts/index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: quickstarts/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/quickstarts/index.adoc description: Use your preferred tool to spin up a local instance of Redpanda Connect and get hands-on experience building your first pipelines. page-git-created-date: "2024-11-14" page-git-modified-date: "2024-11-14" --- Use your preferred tool to spin up a local instance of Redpanda Connect and get hands-on experience building your first pipelines. - [Get Started with Redpanda Connect using `rpk`](rpk/) Deploy your first pipelines using Redpanda Connect and `rpk`. - [Get Started with the Redpanda Connect Helm Chart](helm-chart/) Deploy Redpanda Connect on Kubernetes using Helm. --- # Page 371: Get Started with the Redpanda Connect Helm Chart **URL**: https://docs.redpanda.com/connect/get-started/quickstarts/helm-chart.md --- # Get Started with the Redpanda Connect Helm Chart --- title: Get Started with the Redpanda Connect Helm Chart latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: quickstarts/helm-chart page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: quickstarts/helm-chart.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/quickstarts/helm-chart.adoc description: Deploy Redpanda Connect on Kubernetes using Helm. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-09-22" --- This guide explains how to deploy and configure Redpanda Connect on Kubernetes using the Helm chart. It covers the available deployment modes and walks you through building your own pipelines. If you’re new to Redpanda Connect, see [Introduction to Redpanda Connect](https://docs.redpanda.com/connect/get-started/about/). ## [](#prerequisites)Prerequisites - A running Kubernetes cluster. - The [`kubectl` CLI](https://kubernetes.io/docs/tasks/tools/) and the [`helm` CLI](https://helm.sh/docs/intro/install/) installed. You can optionally provide an enterprise license to enable enterprise connectors. If you have a license, see [Apply a license key to Redpanda Connect](https://docs.redpanda.com/connect/get-started/licensing/) before installing. ## [](#install-with-helm)Install with Helm To deploy Redpanda Connect on Kubernetes, use the official Helm chart. To install the chart: ```bash helm repo add redpanda https://charts.redpanda.com (1) helm repo update (2) helm install redpanda-connect redpanda/connect --namespace --create-namespace (3) ``` | 1 | Adds the Redpanda Helm repository. | | --- | --- | | 2 | Updates your local Helm repository cache. | | 3 | Installs Redpanda Connect in the given namespace. You can customize this deployment by configuring values in the Helm chart. | ## [](#deployment-modes)Deployment modes Redpanda Connect can run in two different modes: **standalone mode** and **streams mode**. - **Standalone mode**: Allows you to run a single pipeline at a time, making it suitable for simpler configurations that don’t need to run concurrently. - **Streams mode**: Supports multiple pipelines running simultaneously, with each pipeline managed through a ConfigMap. Streams mode is ideal for complex data processing use cases. ## [](#run-pipelines-in-standalone-mode)Run pipelines in standalone mode In standalone mode, you can configure a single pipeline within the `config` section of your Helm values file. ### [](#hello-world)Hello world In this example, you’ll produce a simple message and convert it to uppercase using a Bloblang method. 1. Create a `pipeline.yaml` file. This file will be used to override the default Helm chart values for Redpanda Connect. 2. Add the following configuration to `pipeline.yaml`: ```yaml config: input: generate: mapping: root.message = "Hello, Redpanda Connect!" (1) count: 1 (2) pipeline: processors: - mapping: root.message = this.message.uppercase() (3) output: stdout: {} (4) ``` | 1 | Use Bloblang to create a JSON object with the message "Hello, Redpanda Connect!" stored in root.message. | | --- | --- | | 2 | Generate the message only once. | | 3 | Convert the message field in the input to uppercase using the uppercase() method in Bloblang. | | 4 | Output the processed data to stdout, making it viewable in logs. | 3. Deploy the pipeline: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 4. Check the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace ``` You should see the message converted to uppercase in the output: ```json {"message":"HELLO, REDPANDA CONNECT!"} ``` 5. Check the Pod’s status: ```bash kubectl get pods --namespace -l app.kubernetes.io/name=redpanda-connect --watch ``` The Pod enters a `CrashLoopBackOff` state because containers are expected to run continuously. When Redpanda Connect finishes processing the pipeline, the Pod exits, causing Kubernetes to restart it repeatedly. To prevent this status, you can configure Redpanda Connect to continue processing data indefinitely. ### [](#produce-continuous-data)Produce continuous data To produce data continuously, you can set `input.generate.count` to `0`. 1. Update the `pipeline.yaml` file to produce a message every second, indefinitely: ```yaml config: input: generate: interval: 1s count: 0 # Setting count to 0 ensures it generates data indefinitely. mapping: | root.message = "Hello, Redpanda Connect!" pipeline: processors: - mapping: | root.message = this.message.uppercase() output: stdout: {} ``` 2. Deploy the updated configuration: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 3. Watch the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace -f ``` You should see in the logs that Redpanda Connect is producing the same message every second and it’s being converted to uppercase: ```json {"message": "HELLO, REDPANDA CONNECT!"} {"message": "HELLO, REDPANDA CONNECT!"} {"message": "HELLO, REDPANDA CONNECT!"} ``` 4. Check the Pod’s status: ```bash kubectl get pods --namespace -l app.kubernetes.io/name=redpanda-connect --watch ``` The Pod should now be running without entering a `CrashLoopBackOff` state, as the `generate` input continuously feeds new data to the pipeline, preventing it from terminating. ### [](#simulate-realistic-data-streams)Simulate realistic data streams To make the output more realistic, use some Bloblang functions to generate varied data such as random names and emails. 1. Update the `pipeline.yaml` file to generate some realistic user data. ```yaml config: input: generate: interval: 1s count: 0 mapping: | # Store the generated names in variables let first_name = fake("first_name") let last_name = fake("last_name") # Build the message root.user_id = counter() root.name = ($first_name + " " + $last_name) root.timestamp = now() pipeline: processors: - mapping: | root.name = this.name.uppercase() output: stdout: {} ``` This configuration generates a JSON object with: - `user_id`: A unique identifier for each record, generated using the [`counter()`](https://docs.redpanda.com/connect/guides/bloblang/functions/#counter) function. - `name`: A randomly generated first and last name, using the [`fake()`](https://docs.redpanda.com/connect/guides/bloblang/functions/#fake) function. The first and last names are stored in variables and referenced using the `$` syntax. - `timestamp`: The current timestamp at the time of generation, using the [`now()`](https://docs.redpanda.com/connect/guides/bloblang/functions/#now) function. 2. Deploy the updated configuration: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 3. Watch the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace -f ``` You should see logs showing JSON objects similar to the following, with names in uppercase: ```json {"name":"ZOIE SIPES"} {"name":"LORENA KERTZMANN"} {"name":"DALLAS BOYER"} {"name":"LOUIE WILDERMAN"} {"name":"EMILIA KOEPP"} {"name":"KALEIGH PACOCHA"} ``` ### [](#process-data-from-a-file-input)Process data from a file input To configure a pipeline that reads data from a file, first store the data in a ConfigMap. This ConfigMap will be mounted into the Redpanda Connect Pod, allowing it to read the file directly. 1. Create a ConfigMap to provide the input data that Redpanda Connect will read. This example ConfigMap contains a JSON object with example user data: ```bash kubectl create configmap connect-input --from-literal=input-data='{"name": "Redpanda Connect", "email": "rp.connect@example.com"}' --namespace ``` This ConfigMap will act as the source for the file-based input in Redpanda Connect, allowing the pipeline to read and process this structured JSON data. 2. Update the `pipeline.yaml` file to read data from the file mounted by the ConfigMap: `pipeline.yaml` ```yaml extraVolumes: - name: input-config configMap: name: connect-input extraVolumeMounts: - name: input-config mountPath: /input (1) subPath: input-data config: input: file: paths: - "/input" (2) pipeline: processors: - mapping: | root.name = this.name.uppercase() output: stdout: {} ``` | 1 | Mount the ConfigMap data at this path inside the Pod. | | --- | --- | | 2 | Read the input from the mounted file. | 3. Deploy the pipeline: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 4. Check the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace ``` You should see the username converted to uppercase in the output: ```json {"name":"REDPANDA CONNECT"} ``` ## [](#run-multiple-pipelines-in-streams-mode)Run multiple pipelines in streams mode In streams mode, each pipeline, defined in separate YAML files, runs simultaneously, making this mode ideal for high-throughput applications. All the YAML files must be bundled together into a ConfigMap that you can pass to Redpanda Connect. 1. Define your pipeline configurations in the following separate YAML files: `woof.yaml` ```yaml input: generate: mapping: root = "woof" # Generates a message with the word "woof" at regular intervals. interval: 5s count: 0 output: stdout: codec: lines # Outputs each message as a new line in stdout. ``` `meow.yaml` ```yaml input: generate: mapping: root = "meow" # Generates a message with the word "meow" at regular intervals. interval: 2s count: 0 output: stdout: codec: lines # Outputs each message as a new line in stdout. ``` 2. Bundle the configuration files into a ConfigMap, which Redpanda Connect will reference: ```bash kubectl create configmap connect-streams --from-file=woof.yaml --from-file=meow.yaml --namespace ``` 3. Configure Redpanda Connect in streams mode and specify the name of the ConfigMap to use: `connect.yaml` ```yaml streams: enabled: true (1) streamsConfigMap: "connect-streams" (2) ``` | 1 | Enable streams mode in Redpanda Connect. | | --- | --- | | 2 | Use the given ConfigMap as the pipeline configuration. | 4. Deploy the chart: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values connect.yaml ``` 5. Watch the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace -f ``` You should see logs showing a combination of outputs from both pipelines: woof meow meow meow woof meow meow ### [](#update-the-pipeline-in-streams-mode)Update the pipeline in streams mode To update a pipeline in streams mode: 1. Modify one of the configuration files locally. `woof.yaml` ```yaml # Updated woof.yaml input: generate: mapping: root = "bark" # Updated to generate a message with the word "bark" instead of "woof." interval: 5s count: 0 output: stdout: codec: lines ``` 2. Update the ConfigMap with the modified file: ```bash kubectl create configmap connect-streams --from-file=woof.yaml --from-file=meow.yaml --namespace --dry-run=client -o yaml | kubectl apply -f - ``` 3. Restart the Deployment: ```bash kubectl rollout restart deployment/redpanda-connect --namespace ``` ### [](#global-configuration)Global configuration When deploying Redpanda Connect in streams mode, you can configure global tracing, logging, and HTTP settings to apply across all pipelines. Specify these in your `values.yaml` overrides under the `metrics`, `logger`, and `tracing` sections. ```yaml metrics: prometheus: {} # Enable Prometheus metrics collection. tracing: openTelemetry: http: [] # Configure OpenTelemetry HTTP tracing. grpc: [] tags: {} logger: level: INFO # Set logging level (e.g., INFO, DEBUG). static_fields: '@service': redpanda-connect # Add static fields to logs for better traceability. ``` ## [](#access-the-http-server-on-redpanda-connect)Access the HTTP server on Redpanda Connect To manage and monitor Redpanda Connect, you can use its HTTP server, which provides useful endpoints for version checking, pipeline management, and more. By default, Redpanda Connect exposes this server using a Kubernetes ClusterIP Service, accessible only within the cluster. 1. Forward the ports of the ClusterIP Service to your local device: ```bash kubectl port-forward svc/redpanda-connect 8080:80 --namespace ``` 2. Access the HTTP server locally. For example, to check the Redpanda Connect version, run: ```bash curl http://localhost:8080/version ``` Example output: ```json { "version": "v4.38.0", "built": "2024-10-17T09:27:42Z" } ``` You can also configure external access using a LoadBalancer Service or an Ingress. See the [Helm values](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/values.yaml#L79C1-L107C31) for more details. ## [](#uninstall-redpanda-connect)Uninstall Redpanda Connect To remove Redpanda Connect and all related resources from your Kubernetes cluster, use the [`helm uninstall`](https://helm.sh/docs/helm/helm_uninstall/) command to uninstall the chart: ```bash helm uninstall redpanda-connect --namespace ``` This command deletes all resources created by the Helm chart, including Deployments and Services. Uninstalling the chart does not delete the ConfigMaps that you manually created outside of the Helm chart. To delete these ConfigMaps, do the following: ```bash kubectl delete configmap connect-streams connect-input --namespace ``` ## [](#next-steps)Next steps - Learn more about [Bloblang](https://docs.redpanda.com/connect/guides/bloblang/walkthrough/), the mapping language for processing data in Redpanda Connect. - Try more hands-on examples with one of the [Cookbooks](https://docs.redpanda.com/connect/cookbooks/). ## [](#suggested-reading)Suggested reading - [Install or Upgrade with the Helm Chart](https://docs.redpanda.com/connect/install/helm-chart/) - [Streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/) - [Inputs](https://docs.redpanda.com/connect/components/inputs/about/) - [Processors](https://docs.redpanda.com/connect/components/processors/about/) - [Outputs](https://docs.redpanda.com/connect/components/outputs/about/) - [HTTP server](https://docs.redpanda.com/connect/components/http/about/) - [Helm values](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/values.yaml) --- # Page 372: Get Started with Redpanda Connect using rpk **URL**: https://docs.redpanda.com/connect/get-started/quickstarts/rpk.md --- # Get Started with Redpanda Connect using `rpk` --- title: Get Started with Redpanda Connect using rpk latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: quickstarts/rpk page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: quickstarts/rpk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/quickstarts/rpk.adoc description: Deploy your first pipelines using Redpanda Connect and rpk. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-06-16" --- This guide explains how to get started with Redpanda Connect using `rpk`, the Redpanda command-line interface (CLI). You can also install and run `rpk` in [FIPS compliance mode](#fips-compliance). ## [](#install)Install The `rpk` CLI allows you to create and manage data pipelines with Redpanda Connect as well as [interact with Redpanda clusters](https://docs.redpanda.com/streaming/current/get-started/rpk-install/). > 📝 **NOTE** > > The `rpk` CLI also includes an `rpk connect` plugin, which manages installations and upgrades of Redpanda Connect. This plugin is automatically installed when you first run `rpk connect` commands, unless you run `rpk connect --version`, which prompts you to install the plugin. Also interacting with a Redpanda cluster? If you want to use `rpk` to also communicate with a Redpanda cluster, ensure the version of `rpk` that you install matches the version of Redpanda running in your cluster. ### [](#linux)Linux To install, or update to, the latest version of `rpk` for Linux, run: #### amd64 ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-linux-amd64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && sha256sum --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-linux-arm64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && sha256sum --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. To install, or update to, a version other than the latest, run the following command. It verifies the archive against the published SHA-256 checksums before unzipping it: #### amd64 ```bash curl -fLO https://rpk.redpanda.com/v/rpk-linux-amd64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && sha256sum --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -fLO https://rpk.redpanda.com/v/rpk-linux-arm64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && sha256sum --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` #### [](#fips-compliance)FIPS compliance > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. To install `rpk` to run the latest version of Redpanda Connect in FIPS-compliant mode, you must install the `redpanda-rpk-fips` and `redpanda-connect-fips` packages. Both packages are built using the [Microsoft GoLang compiler](https://github.com/microsoft/go) and the [Microsoft’s Go Crypto OpenSSL package](https://github.com/microsoft/go-crypto-openssl), which uses the FIPS-approved version of OpenSSL. > ❗ **IMPORTANT** > > The packages for FIPS-compliant `rpk` (`redpanda-rpk-fips`) and Redpanda `rpk` (`redpanda-rpk`) are mutually exclusive, and so cannot be installed in the same environment. ##### RHEL 1. To make sure your distribution is up to date, run: ```bash sudo dnf upgrade ``` 2. Add `redpanda` to your `dnf` list of repositories. ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo dnf install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with Redpanda Connect releases, run the following command: ```bash sudo dnf update ``` ##### Debian/Ubuntu 1. To make sure your distribution is up to date, run: ```bash sudo apt upgrade ``` 2. Add `redpanda` to your `apt` list of repositories. ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo apt install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with the Redpanda Connect releases, run the following command: ```bash sudo apt update ``` ### [](#macos)MacOS #### Homebrew 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` #### Manual Download To install or update `rpk` through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, select **Apple Silicon**. ##### Intel macOS To install, or update to, the latest version of `rpk` for Intel macOS, run: ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-darwin-amd64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && shasum -a 256 --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run the following command. It verifies the archive against the published SHA-256 checksums before unzipping it: ```bash curl -fLO https://rpk.redpanda.com/v/rpk-darwin-amd64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && shasum -a 256 --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` ##### Apple Silicon To install, or update to, the latest version of `rpk` for Apple Silicon, run: ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-darwin-arm64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && shasum -a 256 --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run the following command. It verifies the archive against the published SHA-256 checksums before unzipping it: ```bash curl -fLO https://rpk.redpanda.com/v/rpk-darwin-arm64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && shasum -a 256 --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` ### [](#docker)Docker To use `rpk` in Docker, you can use the Redpanda Docker image, which includes both Redpanda and `rpk` as part of the installation. ```bash docker pull docker.redpanda.com/redpandadata/connect ``` ## [](#run-a-pipeline)Run a pipeline A Redpanda Connect stream pipeline is configured with a single [config file](https://docs.redpanda.com/connect/configuration/about/), you can generate a fresh one with: ```bash rpk connect create > connect.yaml ``` > 📝 **NOTE** > > This command may take a few seconds to run. If this is the first `rpk connect` command you have run, the `rpk connect plugin` is automatically installed. For Docker installations: ```bash docker run --rm docker.redpanda.com/redpandadata/connect create > ./connect.yaml ``` The main sections that make up a config are `input`, `pipeline` and `output`. When you generate a fresh config it’ll simply pipe `stdin` to `stdout` like this: ```yaml input: stdin: {} pipeline: processors: [] output: stdout: {} ``` Eventually we’ll want to configure a more useful [input](https://docs.redpanda.com/connect/components/inputs/about/) and [output](https://docs.redpanda.com/connect/components/outputs/about/), but for now this is useful for quickly testing processors. You can execute this config with: ```bash rpk connect run connect.yaml ``` For Docker installations: ```bash docker run --rm -it -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run ``` Anything you write to stdin will get written unchanged to stdout! Resist the temptation to play with this for hours, there’s more stuff to try out. Next, let’s add some processing steps in order to mutate messages. The most powerful one is the [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) which allows us to perform mappings, let’s add a mapping to uppercase our messages: ```yaml input: stdin: {} pipeline: processors: - mapping: root = content().uppercase() output: stdout: {} ``` Now your messages should come out in all caps. You can add as many [processing steps](https://docs.redpanda.com/connect/components/processors/about/) as you like, and since processors are what make Redpanda Connect powerful they are worth experimenting with. Let’s create a more advanced pipeline that works with JSON documents: ```yaml input: stdin: {} pipeline: processors: - sleep: duration: 500ms - mapping: | root.doc = this root.first_name = this.names.index(0).uppercase() root.last_name = this.names.index(-1).hash("sha256").encode("base64") output: stdout: {} ``` First, we sleep for 500 milliseconds just to keep the suspense going. Next, we restructure our input JSON document by nesting it within a field `doc`, we map the upper-cased first element of `names` to a new field `first_name`. Finally, we map the hashed and base64 encoded value of the last element of `names` to a new field `last_name`. Try running that config with some sample documents: ```bash echo '{"id":"1","names":["celine","dion"]} {"id":"2","names":["chad","robert","kroeger"]}' | rpk connect run connect.yaml ``` For Docker installations: ```bash echo '{"id":"1","names":["celine","dion"]} {"id":"2","names":["chad","robert","kroeger"]}' | docker run --rm -i -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run ``` You should see this output in the logs: ```json {"doc":{"id":"1","names":["celine","dion"]},"first_name":"CELINE","last_name":"1VvPgCW9sityz5XAMGdI2BTA7/44Wb3cANKxqhiCo50="} {"doc":{"id":"2","names":["chad","robert","kroeger"]},"first_name":"CHAD","last_name":"uXXg5wCKPjpyj/qbivPbD9H9CZ5DH/F0Q1Twytnt2hQ="} ``` See also: - [Bloblang Walkthrough](https://docs.redpanda.com/connect/guides/bloblang/walkthrough/) - [Inputs](https://docs.redpanda.com/connect/components/inputs/about/) - [Processors](https://docs.redpanda.com/connect/components/processors/about/) - [Outputs](https://docs.redpanda.com/connect/components/outputs/about/) - [Monitoring](https://docs.redpanda.com/connect/guides/monitoring/) - [Cookbooks](https://docs.redpanda.com/connect/cookbooks/) - [More about configuration](https://docs.redpanda.com/connect/configuration/about/) --- # Page 373: Redpanda Connect Versioning **URL**: https://docs.redpanda.com/connect/get-started/versioning.md --- # Redpanda Connect Versioning --- title: Redpanda Connect Versioning latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: versioning page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: versioning.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/versioning.adoc description: Learn how Redpanda Connect implements versioning. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-05-26" --- Redpanda Connect uses semantic versioning. Version numbers follow the format `major.minor.patch`: - `major`: This version may include breaking changes. - `minor`: This version only includes backward-compatible new features or enhancements. - `patch`: This version includes backward-compatible bug fixes. For example, version 4.35.1 is the first patch release of bug fixes following the minor release of product enhancements in 4.35.0. Breaking changes would not be released until 5.0.0. ## [](#next-steps)Next steps To benefit from Redpanda Connect’s new features and enhancements, [install the latest version](https://docs.redpanda.com/connect/install/). Learn more about the [versioning of other Redpanda Streaming products](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/). --- # Page 374: What’s New in Redpanda Connect **URL**: https://docs.redpanda.com/connect/get-started/whats-new.md --- # What’s New in Redpanda Connect --- title: What’s New in Redpanda Connect latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: whats-new page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: whats-new.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/whats-new.adoc description: Summary of new features in Redpanda Connect. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-10-15" --- This topic includes new content added from version 4.29.0 onwards. For a full list of product updates, and details of updates prior to version 4.29.0, see the [Redpanda Connect changelog](https://github.com/redpanda-data/connect/blob/main/CHANGELOG.md). For new Cloud-specific features, see [What’s New in Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/get-started/whats-new-cloud/). ## [](#version-4-113-1)Version 4.113.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.113.1). ### [](#component-updates)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | sap_hana | Inputs | stable | Reads rows from a SAP HANA table. | ## [](#version-4-113-0)Version 4.113.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.113.0). ### [](#new-field-support)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | credentials.ssh_algorithms | Additional SSH transport algorithms to permit for compatibility with legacy SFTP servers. Configured algorithms are appended to the default algorithms, which remain preferred, and when unset the defaults are used unchanged. | Inputs:sftpOutputs:sftp | 4.113.0 | ### [](#deprecations)Deprecations The following components are now deprecated: | Component | Type | Description | | --- | --- | --- | | kafka | Outputs | Use the redpanda output instead. The corresponding input was deprecated earlier. | | kafka_franz | Outputs | Use the redpanda output instead. The corresponding input was deprecated earlier. | ### [](#default-value-changes)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | key | ${!count("items")}-${!timestamp_unix_nano()} | ${!timestamp_unix_nano()}-${!uuid_v4()} | The key to store messages by, function interpolation should be used in order to derive a unique key for each message. | Outputs:cache | ## [](#version-4-112-0)Version 4.112.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.112.0). ### [](#new-field-support-2)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | compress_request_body | Enable gzip compression of HTTP request bodies on the OpenSearch client. | Outputs:opensearch | 4.112.0 | ## [](#version-4-111-1)Version 4.111.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.111.1). ### [](#new-field-support-3)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | logminer.window_strategy | Controls how the SCN range mined per cycle is sized: by an adaptive SCN window (scn_window, the default) or by a fixed redo-volume budget (redo_volume). | Inputs:oracledb_cdc | 4.111.1 | | logminer.redo_volume_min | Whilst not exact, this value represents the minimum number of redo logs to read per redo thread in each mining cycle. Consider increasing this value if redo logs are small and rotate frequently, decreasing if redo logs are very large. | Inputs:oracledb_cdc | 4.111.1 | | logminer.redo_volume_growth_max | The ceiling the per-thread redo-volume budget can grow to, applied independently to each open redo thread (enabling the total volume mined per cycle to scale with thread count on RAC (Real Application Clusters) configurations). | Inputs:oracledb_cdc | 4.111.1 | | incremental_snapshot | Configures chunked snapshotting that runs alongside replication streaming. | Inputs:pg_streampostgres_cdc | 4.111.1 | | histogram_timing_seconds_suffix | Whether to rewrite a _ns timing metric name suffix to _seconds when use_histogram_timing is true (for example processor_latency_ns becomes processor_latency_seconds), reflecting that histogram timings are recorded in seconds. | Metrics:prometheus | 4.111.1 | ## [](#version-4-111-0)Version 4.111.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.111.0). ### [](#new-field-support-4)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | bind_http | Whether to register the REST HTTP endpoints (/inputs, /inputs/{id}, and so on) for creating, updating and removing inputs at runtime. These endpoints are served on the service-wide HTTP server and accept input configuration over HTTP, so they are disabled by default. | Inputs:dynamicOutputs:dynamic | 4.111.0 | | prefetch_rows | The number of rows fetched per network round-trip, for both snapshot and streaming reads. Higher values mean fewer round-trips but more memory per fetch, for each table snapshotted in parallel. | Inputs:oracledb_cdc | 4.111.0 | ## [](#version-4-110-0)Version 4.110.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.110.0). ## [](#version-4-109-0)Version 4.109.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.109.0). ### [](#new-field-support-5)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | poll_period | An optional minimum period between GetRecords calls made against each shard. | Inputs:aws_kinesis | 4.109.0 | | enhanced_fan_out | Consume the stream using enhanced fan-out, which provides this consumer dedicated read throughput of 2MB/s per shard via HTTP/2 push delivery, avoiding the 5 reads per second per shard limit that polling consumers share. The named consumer is registered on each stream automatically if it does not already exist (and is never deregistered). | Inputs:aws_kinesis | 4.109.0 | | parquet.compression | The compression codec for data files this output writes. | Outputs:iceberg | 4.109.0 | | max_decompressed_block_bytes | The maximum size, in bytes, that a single compressed Avro OCF block is allowed to expand to when decompressed. This guards against decompression-amplification ("deflate bomb") inputs, where a tiny compressed block would otherwise expand into a very large allocation. | Scanners:avro | 4.109.0 | ## [](#version-4-108-0)Version 4.108.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.108.0). ### [](#new-field-support-6)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | avro.input_encoding | How the values in each message should be read. | Processors:schema_registry_encode | 4.108.0 | ### [](#default-value-changes-2)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | max_message_size | 0 | 33554432 | A maximum size in bytes for individual messages received from the server. When a message exceeding this limit is received the connection is closed with a 1009 (message too big) status and the input reconnects. | Inputs:websocket | ## [](#version-4-107-2)Version 4.107.2 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.107.2). ## [](#version-4-107-1)Version 4.107.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.107.1). ## [](#version-4-107-0)Version 4.107.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.107.0). ### [](#new-field-support-7)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | checkpoint_limit | The maximum number of messages that can be processed at a given time per partition. Increasing this limit enables parallel processing and batching at the output level. | Inputs:gcp_spanner_cdc | 4.107.0 | ## [](#version-4-106-0)Version 4.106.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.106.0). ### [](#new-field-support-8)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | aws | AWS IAM authentication using the MONGODB-AWS mechanism, for example against MongoDB Atlas. When enabled, IAM credentials are used instead of a static username and password. | Caches:mongodbInputs:mongodbmongodb_cdcOutputs:mongodbProcessors:mongodb | 4.106.0 | | auto_replay_nacks | Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. | Inputs:aws_dynamodb_cdc | 4.106.0 | | checkpoint_write_timeout | Bounds the checkpoint writes that run outside the normal read loop - storing the position a completed snapshot reached, and clearing a position that can no longer be resumed from - so that a shutdown racing either of them is not extended indefinitely by a slow cache. | Inputs:mongodb_cdc | 4.106.0 | | on_unresumable_position | What to do when the stored stream position can no longer be resumed from (for example it has aged out of the oplog) and stream_snapshot is disabled, so there is no snapshot to recover with: fail stops the input with an error, preserving the checkpoint for inspection; reset clears the checkpoint and restarts streaming from the current oplog position, skipping the changes between the lost position and now. | Inputs:mongodb_cdc | 4.106.0 | | max_message_size | An optional maximum size in bytes for individual messages received from the server. When a message exceeding this limit is received the connection is closed with a 1009 (message too big) status and the input reconnects. | Inputs:websocket | 4.106.0 | ## [](#version-4-105-0)Version 4.105.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.105.0). ### [](#new-field-support-9)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | signal_table_name | The name of the table used to send control signals to the connector, excluding the schema. | Inputs:pg_streampostgres_cdc | 4.105.0 | | merge_strategy | How upsert and delete are materialised on disk. | Outputs:iceberg | 4.105.0 | | commit.cleanup_on_failure | Whether to remove the files a failed commit had already written. | Outputs:iceberg | 4.105.0 | ## [](#version-4-104-0)Version 4.104.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.104.0). ## [](#version-4-103-2)Version 4.103.2 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.103.2). ### [](#new-field-support-10)New field support This release adds support for the following new fields: | Field | Description | Affected components | Introduced in | | --- | --- | --- | --- | | sqs.zero_key_warn_interval | A message from which no target key can be extracted (for example due to a misconfigured key_path/bucket_path) is never deleted, so it’s redelivered and re-evaluated repeatedly until the underlying issue is fixed. This field limits how often that condition is logged as a warning, to avoid flooding the logs; every occurrence is still logged at debug level. | Inputs:aws_s3 | 4.103.2 | | logminer.max_session_age | The maximum duration a single LogMiner session may stay open before being forcibly ended and restarted, even if the underlying redo log files haven’t changed. By default, a LogMiner session is only restarted when a redo log switch is detected. | Inputs:oracledb_cdc | 4.103.2 | ## [](#version-4-103-1)Version 4.103.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.103.1). ## [](#version-4-103-0)Version 4.103.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.103.0). ## [](#version-4-102-0)Version 4.102.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.102.0). ### [](#new-field-support-11)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | headers | Custom headers to add to migrated records, keyed by header name with interpolated string values. Useful for injecting metadata such as processing timestamps or latency measurements that should surface as header values on the destination cluster. | Outputs:redpanda_migrator | | use_batch | Whether to execute all messages in a batch as a single BatchExecuteStatement call. | Processors:aws_dynamodb_partiql | ## [](#version-4-101-0)Version 4.101.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.101.0). ### [](#new-field-support-12)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | checkpoint_namespace | An optional namespace for checkpoints, allowing multiple independent pipelines (for example one per developer or environment) to share a single checkpoint table without overwriting each other’s positions. | Inputs:aws_dynamodb_cdc | | snapshot_filters | A map of fully-qualified table names (for example SCHEMA.TABLE) to SQL SELECT queries that override the default snapshot query for each table. | Inputs:oracledb_cdc | ## [](#version-4-100-0)Version 4.100.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.100.0). ### [](#component-updates-2)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | jira | Inputs | stable | Periodically queries Jira’s REST API using a JQL filter and emits one message per resource. | ### [](#deprecations-2)Deprecations The following components are now deprecated: | Component | Type | Description | | --- | --- | --- | | jira | Processors | Executes Jira API queries based on input messages and returns structured results. | ## [](#version-4-99-0)Version 4.99.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.99.0). ### [](#new-field-support-13)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | global_table | Provision the checkpoint table as a DynamoDB Global Table (v2) so checkpoints replicate across regions. | Inputs:aws_dynamodb_cdc | | global_table_replicas | Regions other than this pipeline’s own region to replicate the checkpoint table to. The pipeline’s own region is always included. | Inputs:aws_dynamodb_cdc | | snapshot_mode | Controls snapshot behavior. | Inputs:oracledb_cdc | | row_operation | The row-level operation to apply for each message: insert (append), upsert (replace rows matching identifier_fields, then append), or delete (remove rows matching identifier_fields). Supports interpolation so the operation can be driven by the data, such as a change-data-capture stream’s operation field. | Outputs:iceberg | | identifier_fields | The columns forming the row identity (the Iceberg identifier fields / equality-delete key) used by upsert and delete. | Outputs:iceberg | ### [](#deprecations-3)Deprecations The following fields are now deprecated: | Field | Description | Affected components | | --- | --- | --- | | stream_snapshot | Controls whether the connector snapshots existing data before streaming changes. Use snapshot_mode instead. | Inputs:oracledb_cdc | ## [](#version-4-98-0)Version 4.98.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.98.0). ### [](#component-updates-3)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | error_handling | Config | stable | Configures engine-wide error handling behavior. | | try_catch | Processors | beta | Combines the behavior of the try and catch processors into a single block with an explicit recovery path. | ## [](#version-4-97-0)Version 4.97.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.97.0). ### [](#new-field-support-14)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | input_type | Specifies the type of input passed to the model. Required by Cohere embedding models; ignored by Amazon Titan models. | Processors:aws_bedrock_embeddings | | json | Configuration for how to decode schemas that are of type JSON. | Processors:schema_registry_decode | ## [](#version-4-96-3)Version 4.96.3 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.96.3). ## [](#version-4-96-2)Version 4.96.2 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.96.2). ### [](#new-field-support-15)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | api_key | An API key to authenticate with. If set, it supersedes basic authentication. | Outputs:elasticsearch_v8elasticsearch_v9 | ## [](#version-4-96-1)Version 4.96.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.96.1). ## [](#version-4-96-0)Version 4.96.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.96.0). ### [](#component-updates-4)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | doris_stream_load | Outputs | beta | Each output batch is encoded into a single Doris Stream Load request. The sink first contacts FE and follows the Stream Load redirect to BE before uploading the batch body. | ### [](#new-field-support-16)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | change_type | Bloblang expression resolving to the _CHANGE_TYPE pseudo-column value for each row. | Outputs:gcp_bigquery_write_api | | change_sequence_number | Optional Bloblang expression resolving to the _CHANGE_SEQUENCE_NUMBER pseudo-column value. | Outputs:gcp_bigquery_write_api | | primary_keys | Optional list of primary-key column names. | Outputs:gcp_bigquery_write_api | ### [](#default-value-changes-3)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | schema_resolve_timeout | 5s | 15s | How long a single BigQuery table-metadata fetch can run before being aborted. Coalesced concurrent resolves share one fetch, so this bounds the time a wedged backend can stall every batch routing to the same table. | Outputs:gcp_bigquery_write_api | ## [](#version-4-95-0)Version 4.95.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.95.0). ## [](#version-4-94-1)Version 4.94.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.94.1). ## [](#version-4-94-0)Version 4.94.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.94.0). ### [](#new-field-support-17)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | write_mode | How the output writes to BigQuery. | Outputs:gcp_bigquery_write_api | | auto_create_table | If true and the target table does not exist, the output creates it using the configured schema, time_partitioning, and clustering. AlreadyExists errors from concurrent creators are treated as success. | Outputs:gcp_bigquery_write_api | | schema | Column definitions used by auto_create_table. | Outputs:gcp_bigquery_write_api | | time_partitioning | Optional time-partitioning settings applied during auto_create_table. | Outputs:gcp_bigquery_write_api | | clustering | Optional clustering columns (up to 4) applied during auto_create_table. | Outputs:gcp_bigquery_write_api | ## [](#version-4-93-0)Version 4.93.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.93.0). ### [](#new-field-support-18)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | max_cached_streams | Soft cap on the number of cached streams. When the cache exceeds this size, the least-recently-used stream is evicted. | Outputs:gcp_bigquery_write_api | | schema_resolve_timeout | How long a single BigQuery table-metadata fetch can run before being aborted. Coalesced concurrent resolves share one fetch, so this bounds the time a wedged backend can stall every batch routing to the same table. | Outputs:gcp_bigquery_write_api | | schema_evolution_timeout | Total time budget for a single schema evolution attempt (Metadata + Update across all CAS retries on HTTP 412). Bounds how long the WriteBatch retry loop can be starved by a wedged backend. | Outputs:gcp_bigquery_write_api | ### [](#default-value-changes-4)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | max_in_flight | 64 | 4 | The maximum number of messages to have in flight at a given time. Increase this to improve throughput. | Outputs:gcp_bigquery_write_api | ## [](#version-4-92-0)Version 4.92.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.92.0). ### [](#new-field-support-19)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | acks | The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. | Config:redpandaOutputs:kafka_franzredpandaredpanda_migratorTracers:redpanda | | max_buffered_records | The maximum number of records the client will buffer in memory before blocking. | Config:redpandaOutputs:kafka_franzredpandaredpanda_migratorTracers:redpanda | | max_buffered_bytes | The maximum number of bytes the client will buffer in memory before blocking. | Config:redpandaOutputs:kafka_franzredpandaredpanda_migratorTracers:redpanda | | max_in_flight_requests | The maximum number of produce requests in flight per broker connection. | Config:redpandaOutputs:kafka_franzredpandaredpanda_migratorTracers:redpanda | | record_retries | The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. | Config:redpandaOutputs:kafka_franzredpandaredpanda_migratorTracers:redpanda | | record_delivery_timeout | The maximum time a record can sit in the producer buffer before it is failed. Set to 0s for no timeout. This is evaluated before writing a request or after a produce response. | Config:redpandaOutputs:kafka_franzredpandaredpanda_migratorTracers:redpanda | ## [](#version-4-90-3)Version 4.90.3 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.90.3). ### [](#salesforce-connector-refactoring)Salesforce connector refactoring The `salesforce` processor (introduced in [version 4.85.0](#version-4-85-0)) has been removed and replaced with dedicated input components that provide the same functionality with better separation of concerns. The [salesforce\_sink](https://docs.redpanda.com/connect/components/outputs/salesforce_sink/) output remains available. | Component | Type | Status | Description | | --- | --- | --- | --- | | salesforce | Inputs | stable | Executes SOQL queries against Salesforce and returns the results as messages. | | salesforce_cdc | Inputs | stable | Captures change data from Salesforce objects using the Pub/Sub API. | | salesforce_graphql | Inputs | stable | Executes GraphQL queries against Salesforce and returns the results as messages. | #### [](#migration-guide)Migration guide To migrate from the removed `salesforce` processor, use the appropriate input component based on your use case: | Previous processor feature | Replacement | | --- | --- | | SOQL queries | salesforce input | | Change Data Capture (CDC) | salesforce_cdc input | | GraphQL queries | salesforce_graphql input | ## [](#version-4-90-1)Version 4.90.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.90.1). ## [](#version-4-90-0)Version 4.90.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.90.0). ### [](#component-updates-5)Component updates This release adds the following new component: | Component | Type | Status | Description | | --- | --- | --- | --- | | gcp_bigquery_write_api | Outputs | stable | Writes messages to a BigQuery table using the Storage Write API. This provides higher throughput and lower latency than the legacy streaming API or load jobs. | ### [](#new-field-support-20)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | max_parallel_snapshot_tables | Specifies the number of tables that will be snapshotted in parallel. | Inputs:mysql_cdc | ## [](#version-4-89-3)Version 4.89.3 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.89.3). ### [](#new-field-support-21)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | case_sensitive_columns | Controls how message field names are matched against table column names, and how column references in the partition spec are resolved. | Outputs:iceberg | ## [](#version-4-89-2)Version 4.89.2 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.89.2). ## [](#version-4-89-1)Version 4.89.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.89.1). ## [](#version-4-89-0)Version 4.89.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.89.0). ### [](#new-field-support-22)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | default_timestamp_unit | The precision used when encoding TIMESTAMP logical types. | Processors:parquet_encode | ## [](#version-4-88-0)Version 4.88.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.88.0). ### [](#component-updates-6)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | arc | Outputs | stable | This output sends data to an Arc columnar analytical database using its high-performance MessagePack ingestion endpoint. | | open_telemetry_collector | Metrics | stable | Exports Redpanda Connect metrics to one or more OpenTelemetry Collector endpoints over HTTP or gRPC for aggregation and onward export. | ### [](#new-field-support-23)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | pdb_name | The name of the pluggable database (PDB) to monitor. When connecting to a CDB root, LogMiner output is scoped to this PDB via SRC_CON_NAME filtering and catalog queries use ALTER SESSION SET CONTAINER to switch context. | Inputs:oracledb_cdc | ### [](#deprecations-and-removals)Deprecations and removals The MCP (Model Context Protocol) server feature in Redpanda Connect has been deprecated and removed. This includes: - All MCP server documentation - The following CLI commands: - `rpk connect mcp-server` - `rpk connect mcp-server init` - `rpk connect mcp-server lint` ## [](#version-4-87-0)Version 4.87.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.87.0). ### [](#new-field-support-24)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | checkpoint_cache_connection_string | An optional connection string for a remote Microsoft SQL Server to use for the checkpoint cache. When set, the checkpoint cache table is created on this remote server instead of the source database. | Inputs:microsoft_sql_server_cdc | | wallet_path | Path to the Oracle Wallet directory. When set, SSL is enabled automatically. | Inputs:oracledb_cdc | | wallet_password | Password for the ewallet.p12 PKCS#12 wallet file. Only required when the wallet directory contains ewallet.p12 rather than cwallet.sso. | Inputs:oracledb_cdc | | message_properties_message_id | Set the message-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. | Outputs:amqp_1 | | message_properties_correlation_id | Set the correlation-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. | Outputs:amqp_1 | | message_properties_subject | Set the subject property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_reply_to | Set the reply-to property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_group_id | Set the group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_group_sequence | Set the group-sequence property on outgoing AMQP messages. Must be a valid uint32 value. | Outputs:amqp_1 | | message_properties_reply_to_group_id | Set the reply-to-group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_user_id | Set the user-id property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_content_type | Set the content-type property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_content_encoding | Set the content-encoding property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | ## [](#version-4-86-0)Version 4.86.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.86.0). ### [](#component-updates-7)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | string_split | Processors | stable | Splits a string by a delimiter into an array. Generally, using bloblang’s split method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. | ## [](#version-4-85-0)Version 4.85.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.85.0). ### [](#bloblang-updates)Bloblang updates This release adds the following new Bloblang capabilities: - Methods: - [`escape_url_path`](https://docs.redpanda.com/connect/guides/bloblang/methods/#escape_url_path) Escapes a string so it can be safely placed inside a URL path segment, replacing special characters with percent-encoded sequences. - [`parse_logfmt`](https://docs.redpanda.com/connect/guides/bloblang/methods/#parse_logfmt) Parses a logfmt-encoded string into an object of key-value pairs. - [`unescape_url_path`](https://docs.redpanda.com/connect/guides/bloblang/methods/#unescape_url_path) Unescapes a URL path segment, converting percent-encoded sequences back to their original characters. ### [](#component-updates-8)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | oracledb_cdc | Inputs | Streams changes from an Oracle database for Change Data Capture (CDC). | | salesforce_sink | Outputs | Consumes batches of messages and writes them to Salesforce. | | salesforce | Processors | Fetches data from Salesforce based on input messages. | ### [](#new-field-support-25)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | client_name | Set the client name for the Redis connection. | Caches:redisInputs:redis_listredis_pubsubredis_scanredis_streamsOutputs:redis_hashredis_listredis_pubsubredis_streamsProcessors:redisredis_scriptRate-limits:redis | | tables | List of table names to stream from. For single table mode, provide one table. | Inputs:aws_dynamodb_cdc | | table_discovery_mode | Table discovery mode. | Inputs:aws_dynamodb_cdc | | table_tag_filter | Multi-tag filter: 'key1:v1,v2;key2:v3,v4'. Matches tables with (key1=v1 OR key1=v2) AND (key2=v3 OR key2=v4). | Inputs:aws_dynamodb_cdc | | table_discovery_interval | Interval for rescanning and discovering new tables when using tag or includelist mode. Set to 0 to disable periodic rescanning. | Inputs:aws_dynamodb_cdc | | snapshot_mode | Snapshot behavior. | Inputs:aws_dynamodb_cdc | | snapshot_segments | Number of parallel scan segments (1-10). Higher parallelism scans faster but consumes more RCUs. | Inputs:aws_dynamodb_cdc | | snapshot_batch_size | Records per scan request during snapshot. Maximum 1000. | Inputs:aws_dynamodb_cdc | | snapshot_throttle | Minimum time between scan requests per segment. Use this to limit RCU consumption during snapshot. | Inputs:aws_dynamodb_cdc | | snapshot_deduplicate | Deduplicate records that appear in both snapshot and CDC stream. Requires buffering CDC events during snapshot. | Inputs:aws_dynamodb_cdc | | snapshot_buffer_size | Maximum CDC events to buffer for deduplication (approximately 100 bytes per entry). If exceeded, deduplication is disabled and duplicates may be emitted. | Inputs:aws_dynamodb_cdc | | create_stream | Whether to automatically create the stream if it doesn’t exist (requires the stream field to be set). | Inputs:nats_jetstream | | extract_tracing_map | A Bloblang mapping that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. | Inputs:redpanda | | id | The entry ID for the stream message. Allows function interpolations. | Outputs:redis_streams | | inject_tracing_map | A Bloblang mapping used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. | Outputs:redpanda | | sync_topic_interval | How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow. | Outputs:redpanda_migrator | | commit_backoff | Control how frequently Snowflake is polled to check if data has been committed. | Outputs:snowflake_streaming | | timeout | HTTP request timeout. | Processors:jira | | tls | Custom TLS settings can be used to override system defaults. | Processors:jira | | proxy_url | HTTP proxy URL. Empty string disables proxying. | Processors:jira | | disable_http2 | Disable HTTP/2 and force HTTP/1.1. | Processors:jira | | tps_limit | Rate limit in requests per second. 0 disables rate limiting. | Processors:jira | | tps_burst | Maximum burst size for rate limiting. | Processors:jira | | backoff | Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. | Processors:jira | | tcp | TCP socket configuration. | Processors:jira | | http | HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. | Processors:jira | | access_log_level | Log level for HTTP request/response logging. Empty disables logging. | Processors:jira | | access_log_body_limit | Maximum bytes of request/response body to include in logs. 0 to skip body logging. | Processors:jira | | schema_metadata | When set, the processor reads a schema in benthos common schema format from this metadata key on each message, converts it to the format specified by format, registers it with the schema registry under the configured subject, and encodes the message. When empty (the default), the processor pulls the latest schema from the registry instead. | Processors:schema_registry_encode | | format | The encoding format to use when converting a common schema from metadata. | Processors:schema_registry_encode | | normalize | Whether to normalize the schema before registering with the schema registry (schema_metadata mode only). | Processors:schema_registry_encode | | avro | Configuration for Avro encoding. | Processors:schema_registry_encode | | continue | Indicates whether, if this case passes for a message, the next case should also be tested. | Processors:switch | ### [](#deprecations-4)Deprecations The following fields are now deprecated: | Field | Description | Affected components | | --- | --- | --- | | commit_timeout | Use max_elapsed_time instead. | Outputs:snowflake_streaming | | avro_raw_json | Use avro.raw_json instead. | Processors:schema_registry_encode | ### [](#default-value-changes-5)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | target_address | none | "" | The target address to write to. | Outputs:amqp_1 | | commit_timeout | 60s | "" | Use max_elapsed_time instead. | Outputs:snowflake_streaming | ### [](#removed-components)Removed components The following deprecated components have been removed: - `legacy_redpanda_migrator` input and output - `legacy_redpanda_migrator_offsets` input and output - `redpanda_migrator_bundle` input and output Use the unified [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) input and output instead. ## [](#version-4-81-0)Version 4.81.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.81.0). ### [](#component-updates-9)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | aws_cloudwatch_logs | Input | Consumes log events from AWS CloudWatch Logs. | ### [](#new-field-support-26)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | timeout_seconds | The timeout in seconds, for querying the TigerBeetle cluster. | Inputs:tigerbeetle_cdc | | subject | An optional subject to set for messages. | Outputs:aws_sns | | shared_drives | Whether or not to include shared drives. | Processors:google_drive_downloadgoogle_drive_search | ## [](#version-4-80-0)Version 4.80.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.80.0). ### [](#component-updates-10)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | iceberg | Output | Write data to Apache Iceberg tables via REST catalog. | ## [](#version-4-79-0)Version 4.79.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.79.0). ### [](#component-updates-11)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | aws_dynamodb_cdc | Inputs | Consumes records from DynamoDB Streams with automatic checkpointing and shard management. | ### [](#new-field-support-27)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | encoding | Encoding format for messages in the batch. Options: 'protobuf' or 'json'. | Inputs:otlp_grpcotlp_http | | schema_registry | Optional Schema Registry configuration for adding Schema Registry wire format headers to messages. | Inputs:otlp_grpcotlp_http | | message_format | The format to expect incoming messages from the rest of the pipeline. | Outputs:snowflake_streaming | | timestamp_format | The format to parse string values for TIMESTAMP, TIMESTAMP_LTZ and TIMESTAMP_NTZ columns. Should be a layout for time.Parse in Go. | Outputs:snowflake_streaming | ## [](#version-4-78-0)Version 4.78.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.78.0). ### [](#new-field-support-28)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tcp | TCP listener configuration for the HTTP server. | Inputs:http_server | ## [](#version-4-77-0)Version 4.77.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.77.0). ### [](#component-updates-12)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | elasticsearch_v9 | Outputs | Publish messages into an Elasticsearch index. | ## [](#version-4-76-1)Version 4.76.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.76.1). ## [](#version-4-76-0)Version 4.76.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.76.0). ### [](#component-updates-13)Component updates This release adds the following new components: > 📝 **NOTE** > > Requires a cgo-enabled binary. See the [installation guides](https://docs.redpanda.com/connect/install/) for details. ### [](#new-field-support-29)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tags | Global tags added to each metric. | Metrics:statsd | ## [](#version-4-75-0)Version 4.75.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.75.0). ### [](#new-field-support-30)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | batching | Optional explicit batching policy for the output. Note that when batches are formed at the input level they can be expanded by this policy, but not contracted. | Outputs:redpanda | ## [](#version-4-74-0)Version 4.74.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.74.0). ## [](#version-4-73-0)Version 4.73.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.73.0). ### [](#new-field-support-31)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tcp | TCP listener socket configuration. | Inputs:socket_server | | ttl | An optional TTL to set for items. | Outputs:couchbaseProcessors:couchbase | | schema_registry | Schema registry information to publish schemas for tracing data along with the data. | Tracers:redpanda | ## [](#version-4-72-0)Version 4.72.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.72.0). ## [](#version-4-71-0)Version 4.71.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.71.0). ### [](#component-updates-14)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | redpanda | Tracers | - | ### [](#new-field-support-32)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | aws | AWS IAM authentication configuration for PostgreSQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. | Inputs:pg_streampostgres_cdc | | offset_header | Header name to add to migrated records containing the source offset for exact consumer group migration. If empty, no offset header is added and exact offset translation is disabled. | Outputs:redpanda_migrator | | max_in_flight | Maximum number of batches to have in flight at any given time. For optimal throughput, set this to the total number of partitions being copied in parallel (up to all partitions in the cluster). | Outputs:redpanda_migrator | ## [](#version-4-70-0)Version 4.70.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.70.0). ### [](#bloblang-updates-2)Bloblang updates This release adds the following new Bloblang capabilities: - Functions: - [`bytes`](https://docs.redpanda.com/connect/guides/bloblang/functions/#bytes): Create a new byte array that is zero initialized. - Methods: - [`repeat`](https://docs.redpanda.com/connect/guides/bloblang/methods/#repeat): Creates a new array by repeating the elements of the input array a specified number of times. ### [](#new-field-support-33)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tcp | Configure TCP socket-level settings to optimize network performance and reliability.Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. | Config:redpandaCaches:aws_dynamodbaws_s3redpandaInputs:aws_kinesisaws_s3aws_sqskafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migratorOutputs:aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqskafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migratorProcessors:aws_bedrock_chataws_bedrock_embeddingsaws_dynamodb_partiqlaws_lambdaMetrics:aws_cloudwatch | | regexp_topics_include | A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the metadata_max_age interval. | Inputs:kafka_franzlegacy_redpanda_migratorredpandaredpanda_commonredpanda_migrator | | regexp_topics_exclude | A list of regular expression patterns for excluding topics. Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. | Inputs:kafka_franzlegacy_redpanda_migratorredpandaredpanda_commonredpanda_migrator | | max_parallel_snapshot_tables | The number of tables that will be processed in parallel during the snapshot processing stage. | Inputs:microsoft_sql_server_cdc | | tls | Custom TLS settings to override system defaults. | Inputs:pg_streampostgres_cdc | | persistent | If set to true, the message is marked as persistent, ensuring it is stored durably and not lost if an intermediary (such as a broker) restarts. By default, messages are not durable. | Outputs:amqp_1 | | target_capabilities | Lists the extension capabilities the sender desires from the target, such as support for queues, topics, durability, sharing, or temporary destinations. | Outputs:amqp_1 | | message_properties_to | The node that is the intended destination of the message, which may differ from the node currently receiving the transfer. | Outputs:amqp_1 | | provenance_header | Header name to add to migrated records indicating their source cluster. When set, each migrated message receives a header with this name containing the source cluster’s seed broker addresses, enabling downstream systems to track message origins for auditing, debugging, or multi-cluster orchestration workflows. | Outputs:redpanda_migrator | ### [](#deprecations-5)Deprecations The following fields are now deprecated: | Field | Description | Affected components | | --- | --- | --- | | regexp_topics | This boolean field is deprecated in favor of regexp_topics_include and regexp_topics_exclude, which provide more explicit control over topic pattern matching. To migrate, replace regexp_topics: true with regexp_topics_include containing your topic patterns, and optionally add regexp_topics_exclude to filter out specific topics. For example, if you had topics: ["events_."] with regexp_topics: true, change to regexp_topics_include: ["events_."]. | Inputs:kafka_franzlegacy_redpanda_migratorredpandaredpanda_commonredpanda_migrator | ### [](#default-value-changes-6)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | metadata_max_age | 5m | 1m | The default changed from 5m to 1m to provide more responsive topic and partition discovery. This means metadata refreshes occur more frequently by default. If you experience increased broker load or want to maintain the previous behavior, explicitly set this field to 5m. | Config:redpandaCaches:redpandaInputs:kafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migratorOutputs:kafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migrator | ## [](#version-4-68-0)Version 4.68.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.68.0). ### [](#component-updates-15)Component updates This release adds the following new components: - Processors: - [`jira`](https://docs.redpanda.com/connect/components/processors/jira/): Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically. This release adds support for the following new fields: - Inputs components - [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/): - [`unordered_processing`](https://docs.redpanda.com/connect/components/inputs/redpanda/#unordered_processing) - Configures partition consumers to allow parallel and therefore unordered processing of messages of any given partition. This allows for better utilization of processing threads and asynchronous publishing at the output level. The maximum parallelization of each partition is determined by the checkpoint\_limit field. ### [](#deprecations-6)Deprecations The `kafka`, `kafka_franz`, and `redpanda_common` inputs and outputs are now deprecated. Their functionality has been consolidated into the unified `redpanda` input and output components. Migrate existing configurations to use the new `redpanda` components for continued support and access to the latest features. ## [](#version-4-67-5)Version 4.67.5 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.67.5). ### [](#component-updates-16)Component updates This release adds the following new components: - Inputs: - [`microsoft_sql_server_cdc`](https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc/): Streams change data from a Microsoft SQL Server database into Redpanda Connect using Change Data Capture (CDC). This release adds support for the following new fields: - Inputs components - [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) - [`schema_registry`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/#schema_registry) - Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. - Outputs components - [`redpanda_migrator`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) - [`schema_registry`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/#schema_registry) - Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. - [`consumer_groups`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/#consumer_groups) - Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed. - [`topic_replication_factor`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/#topic_replication_factor) - The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes. - [`sync_topic_acls`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/#sync_topic_acls) - Whether to synchronize topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts. - [`serverless`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/#serverless) - Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by Redpanda Serverless environments. ### [](#deprecations-7)Deprecations This release deprecates the individual migrator components and the bundle wrapper in favor of the unified `redpanda_migrator` architecture. **Deprecated components:** - `redpanda_migrator` input and output (renamed to `legacy_redpanda_migrator`) - `redpanda_migrator_offsets` input and output (renamed to `legacy_redpanda_migrator_offsets`) - `redpanda_migrator_bundle` input and output (now deprecated; these are part of the legacy migration architecture and internally depend on the deprecated `legacy_redpanda_migrator` and `legacy_redpanda_migrator_offsets` components) These components have been renamed from their previous versions (without the `legacy_` prefix) to preserve backward compatibility. The new `redpanda_migrator` components introduced in this release contain breaking changes. Existing configurations using the previous `redpanda_migrator` components should use these `legacy_` prefixed versions to maintain current functionality. Users should migrate from deprecated components to the new unified `redpanda_migrator` input/output pair. For detailed migration instructions, see [Migrate to the Unified Redpanda Migrator](https://docs.redpanda.com/connect/guides/migration/migrate-unified-redpanda-migrator/). ## [](#version-4-66-0)Version 4.66.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.66.0). ### [](#component-updates-17)Component updates This release adds the following new components: - Outputs: - [`cyborgdb`](https://docs.redpanda.com/connect/components/outputs/cyborgdb/): This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization. All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure. - Bloblang-functions: - [`with_schema_registry_header`](https://docs.redpanda.com/connect/guides/bloblang/functions/#with_schema_registry_header): Prepends a 5-byte Schema Registry header to a message. The header consists of a magic byte (0x00) followed by a 4-byte big-endian schema ID. This release adds support for the following new fields: - Inputs components - [`cassandra`](https://docs.redpanda.com/connect/components/inputs/cassandra/) - [`reconnect_interval`](https://docs.redpanda.com/connect/components/inputs/cassandra/#reconnect_interval) - Attempts to reconnect known DOWN nodes in every ReconnectInterval. - [`exponential_reconnection`](https://docs.redpanda.com/connect/components/inputs/cassandra/#exponential_reconnection) - Optional exponential reconnection policy, this replaces the default constant policy of the driver. - Outputs components - [`cassandra`](https://docs.redpanda.com/connect/components/outputs/cassandra/) - [`reconnect_interval`](https://docs.redpanda.com/connect/components/outputs/cassandra/#reconnect_interval) - Attempts to reconnect known DOWN nodes in every ReconnectInterval. - [`exponential_reconnection`](https://docs.redpanda.com/connect/components/outputs/cassandra/#exponential_reconnection) - Optional exponential reconnection policy, this replaces the default constant policy of the driver. ## [](#version-4-65-0)Version 4.65.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.65.0). ### [](#component-updates-18)Component updates This release adds the following new components: - Inputs: - [`tigerbeetle_cdc`](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/): Streams change data from a TigerBeetle database into Redpanda Connect. > 📝 **NOTE** > > The `tigerbeetle_cdc` input requires a CGO-enabled build of Redpanda Connect. > > For instructions, see: > > - [Download a CGO-enabled binary](https://docs.redpanda.com/connect/install/prebuilt-binary/) > > - [Build Redpanda Connect from source](https://docs.redpanda.com/connect/install/build-from-source/) - Scanners: - [`json_array`](https://docs.redpanda.com/connect/components/scanners/json_array/): Scans a JSON array and emits each element as a separate message. ### [](#model-context-protocol-mcp-server)Model Context Protocol (MCP) server You can now build and run an MCP server directly in Redpanda Connect. The MCP server lets you expose your Redpanda Connect configurations as AI-consumable HTTP endpoints, making them discoverable and callable by AI clients such as Claude Code and other compatible agents. ## [](#version-4-64-0)Version 4.64.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.64.0). ### [](#component-updates-19)Component updates This release adds support for the following new fields and improvements: - Schema registry decode processor: - [`default_schema_id`](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/#default_schema_id) field for specifying a fallback schema ID when decoding messages without an explicit schema reference. ## [](#version-4-62-0)Version 4.62.0 ### [](#component-updates-20)Component updates This release adds support for the following new fields and improvements: - Kafka, Redpanda, and Franz input components: - [`timely_nacks_maximum_wait`](https://docs.redpanda.com/connect/components/inputs/kafka/#timely_nacks_maximum_wait) field for improved message delivery guarantees and SLA enforcement. - [`timely_nacks_maximum_wait`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/#timely_nacks_maximum_wait) field for improved message delivery guarantees and SLA enforcement. - [`timely_nacks_maximum_wait`](https://docs.redpanda.com/connect/components/inputs/redpanda/#timely_nacks_maximum_wait) field for improved message delivery guarantees and SLA enforcement. - Redpanda Migrator and Redpanda Common input components: - [`timely_nacks_maximum_wait`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) field for consistent negative acknowledgement handling. - [`timely_nacks_maximum_wait`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/#timely_nacks_maximum_wait) field for consistent negative acknowledgement handling. - Schema Registry output component: - [`subject_compatibility_level`](https://docs.redpanda.com/connect/components/outputs/schema_registry/#subject_compatibility_level) field for explicit subject compatibility configuration. - Parquet Encode processor: - [`schema_metadata`](https://docs.redpanda.com/connect/components/processors/parquet_encode/#schema_metadata) field for dynamic schema assignment during encoding. - Socket output component: - [`Full TLS configuration support`](https://docs.redpanda.com/connect/components/outputs/socket/#tls), including all standard and mTLS fields, now deduplicated and centrally documented. - Bloblang: - [`infer_schema`](https://docs.redpanda.com/connect/guides/bloblang/methods/#infer_schema) method attempts to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods. ## [](#version-4-61-0)Version 4.61.0 ### [](#component-updates-21)Component updates This release adds support for the following new fields: - Cassandra input and output components - Input: [`host_selection_policy`](https://docs.redpanda.com/connect/components/inputs/cassandra/#host_selection_policy) - Output: [`host_selection_policy`](https://docs.redpanda.com/connect/components/outputs/cassandra/#host_selection_policy) - Schema Registry output component - [`normalize`](https://docs.redpanda.com/connect/components/outputs/schema_registry/#normalize) - [`remove_metadata`](https://docs.redpanda.com/connect/components/outputs/schema_registry/#remove_metadata) - [`remove_rule_set`](https://docs.redpanda.com/connect/components/outputs/schema_registry/#remove_rule_set) ## [](#version-4-60-0)Version 4.60.0 ### [](#component-updates-22)Component updates The 4.60.0 release adds support for the following fields: - NATS components - `tls_handshake_first` - Enhanced TLS configuration option for NATS connections that require the initial TLS handshake to be performed before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. This field is available in the following NATS components: - [`nats` input](https://docs.redpanda.com/connect/components/inputs/nats/#tls_handshake_first) - [`nats_jetstream` input](https://docs.redpanda.com/connect/components/inputs/nats_jetstream/#tls_handshake_first) - [`nats_kv` input](https://docs.redpanda.com/connect/components/inputs/nats_kv/#tls_handshake_first) - [`nats_stream` input](https://docs.redpanda.com/connect/components/inputs/nats_stream/#tls_handshake_first) - [`nats` output](https://docs.redpanda.com/connect/components/outputs/nats/#tls_handshake_first) - [`nats_jetstream` output](https://docs.redpanda.com/connect/components/outputs/nats_jetstream/#tls_handshake_first) - [`nats_kv` output](https://docs.redpanda.com/connect/components/outputs/nats_kv/#tls_handshake_first) - [`nats_stream` output](https://docs.redpanda.com/connect/components/outputs/nats_stream/#tls_handshake_first) - [`nats_kv` processor](https://docs.redpanda.com/connect/components/processors/nats_kv/#tls_handshake_first) - [`nats_request_reply` processor](https://docs.redpanda.com/connect/components/processors/nats_request_reply/#tls_handshake_first) - [`nats_kv` cache](https://docs.redpanda.com/connect/components/caches/nats_kv/#tls_handshake_first) ## [](#version-4-59-0)Version 4.59.0 ### [](#redpanda-connect-in-redpanda-cloud-ga)Redpanda Connect in Redpanda Cloud: GA Redpanda Connect is now generally available (GA) in all Redpanda Cloud clusters: BYOC (including BYOVPC/BYOVNet), Dedicated, and Serverless. Redpanda Connect has been GA in Redpanda Streaming clusters since version 4.29.0. ### [](#component-updates-23)Component updates This release adds support for the following fields: - SFTP input components - [`connection_timeout`](https://docs.redpanda.com/connect/components/inputs/sftp/#connection_timeout) - [`credentials.host_public_key`](https://docs.redpanda.com/connect/components/inputs/sftp/#credentials-host_public_key) - [`credentials.host_public_key_file`](https://docs.redpanda.com/connect/components/inputs/sftp/#credentials-host_public_key_file) - [`max_sftp_sessions`](https://docs.redpanda.com/connect/components/inputs/sftp/#max_sftp_sessions) - SFTP output components - [`connection_timeout`](https://docs.redpanda.com/connect/components/outputs/sftp/#connection_timeout) - [`credentials.host_public_key`](https://docs.redpanda.com/connect/components/outputs/sftp/#credentials-host_public_key) - [`credentials.host_public_key_file`](https://docs.redpanda.com/connect/components/outputs/sftp/#credentials-host_public_key_file) - GCP Pub/Sub output components - [`validate_topic`](https://docs.redpanda.com/connect/components/outputs/gcp_pubsub/#validate_topic) - Redpanda cache components - [`allow_auto_topic_creation`](https://docs.redpanda.com/connect/components/caches/redpanda/#allow_auto_topic_creation) ## [](#version-4-58-2)Version 4.58.2 This release introduces an experimental Slack output and expands support for automatic topic creation across Redpanda-related components. ### [](#new-components)New components - [`slack_reaction`](https://docs.redpanda.com/connect/components/outputs/slack_reaction/): Adds or removes emoji reactions from Slack messages by channel ID and timestamp. ### [](#component-updates-24)Component updates This release adds support for the `allow_auto_topic_creation` field in the following components: - [`redpanda`](https://docs.redpanda.com/connect/components/redpanda/about/) configuration block. - [`kafka_franz`](https://docs.redpanda.com/connect/components/outputs/kafka_franz/) output. - [`redpanda`](https://docs.redpanda.com/connect/components/outputs/redpanda/) output. - [`redpanda_migrator`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) output. This field allows components to create topics automatically when publishing data, if the underlying Kafka/Redpanda broker is configured to allow it. ## [](#version-4-57-0)Version 4.57.0 This release adds support for the following fields: - `protobuf.use_proto_names`, `protobuf.use_enum_numbers`, `protobuf.emit_unpopulated`, and `protobuf.emit_default_values` in the [`schema_registry_decode` processor](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/). ## [](#version-4-56-0)Version 4.56.0 This release introduces a dynamic [plugins](https://docs.redpanda.com/connect/plugins/about/) framework. It enables you to create and load custom plugins at runtime using any programming language that supports gRPC, with initial SDKs for Go and Python. ### [](#new-components-2)New components - [`gcp_spanner_cdc`](https://docs.redpanda.com/connect/components/inputs/gcp_spanner_cdc/): Streams change data from a Google Cloud Spanner instance. ### [](#component-updates-25)Component updates This release adds support for the following fields: - `scope` in the [`couchbase`](https://docs.redpanda.com/connect/components/caches/couchbase/) cache. - `poll_interval` and `max_yield_batch_bytes` in the `redpanda_migrator_offsets` input. - `consumer_group_offsets_poll_interval` in the `redpanda_migrator_bundle` input. - `input_bundle_label` in the `redpanda_migrator_bundle` output. - `object_canned_acl` in the [`aws_s3`](https://docs.redpanda.com/connect/components/outputs/aws_s3/) output. - `history`, `max_tool_calls`, and `tools` in the [`gcp_vertex_ai_chat`](https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_chat/) processor. ## [](#version-4-55-1)Version 4.55.1 This release adds support for the following fields: - `is_serverless` in the [`redpanda_migrator`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) output. ## [](#version-4-55-0)Version 4.55.0 This release introduces a new beta Redpanda cache component and new fields across several inputs and outputs. ### [](#new-components-3)New components - [`redpanda`](https://docs.redpanda.com/connect/components/caches/redpanda/): A new cache that stores key/value pairs in a compacted Redpanda topic. ### [](#component-updates-26)Component updates This release adds support for the following fields: - `steal_grace_period` in the [`aws_kinesis`](https://docs.redpanda.com/connect/components/inputs/aws_kinesis/) input. - `max_yield_batch_bytes` in: - [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/) - [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/) - [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) input - [`redpanda_migrator`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) output - `extras` in the [`sentry_capture`](https://docs.redpanda.com/connect/components/processors/sentry_capture/) processor. ### [](#removed-components-2)Removed components The deprecated `elasticsearch` output has been removed. Use the [`elasticsearch_v8`](https://docs.redpanda.com/connect/components/outputs/elasticsearch_v8/) output instead. ### [](#new-bloblang-features)New Bloblang features - [`bitwise_and`](https://docs.redpanda.com/connect/guides/bloblang/methods/#bitwise_and), [`bitwise_or`](https://docs.redpanda.com/connect/guides/bloblang/methods/#bitwise_or), and [`bitwise_xor`](https://docs.redpanda.com/connect/guides/bloblang/methods/#bitwise_xor) methods added for performing bitwise operations on integers. ## [](#version-4-54-0-to-4-54-1)Version 4.54.0 to 4.54.1 These releases introduce the [`qdrant` processor](https://docs.redpanda.com/connect/components/processors/qdrant/), which allows you to query items within a Qdrant collection and filter the returned results. ### [](#component-updates-27)Component updates These releases add: - The `topic_lag_refresh_period` field to the [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/) and [`ockam_kafka`](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/) inputs. - The `cache_duration` field to the [`schema_registry_decode` processor](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/). - The `client_auth` field to the [`socket_server` input](https://docs.redpanda.com/connect/components/inputs/socket_server/). You can also now use Bloblang to generate a [name-based UUID (version 5)](https://docs.redpanda.com/connect/guides/bloblang/methods/#uuid_v5) for a given string. ## [](#version-4-53-0)Version 4.53.0 This release introduces three new Google Drive processors: - [`google_drive_search` processor](https://docs.redpanda.com/connect/components/processors/google_drive_search/): Searches Google Drive for files that match a specified query and emits the results as a batch of messages. - [`google_drive_download` processor](https://docs.redpanda.com/connect/components/processors/google_drive_download/): Downloads files from Google Drive that contain matching file IDs. - [`google_drive_list_labels` processor](https://docs.redpanda.com/connect/components/processors/google_drive_list_labels/): Lists labels for files on a Google Drive. The release also adds the [`cohere_rerank` processor](https://docs.redpanda.com/connect/components/processors/cohere_rerank/), which sends document strings to the Cohere API, and returns them ranked by their relevance to a specified query. ### [](#component-updates-28)Component updates This release adds: - The `transaction_isolation_level` field to the following inputs: [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`ockam_kafka`](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/), [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/). - The `request_timeout_overhead` and `conn_idle_timeout` fields to the following components: [`kafka_franz` input](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/), [`kafka_franz` output](https://docs.redpanda.com/connect/components/outputs/kafka_franz/), [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/), [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/), `redpanda_migrator_offsets` output, [`redpanda` configuration service](https://docs.redpanda.com/connect/components/redpanda/about/). - The `start_offset` field to the following components: [`kafka_franz` input](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`ockam_kafka` input](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/), [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common` input](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/). - The `topic_prefix` field to the [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/), and the `offset_topic_prefix` field to the `redpanda_migrator_offsets` output, to more easily identify migrated topics. - The `use_enum_numbers` field to the [`protobuf` processor](https://docs.redpanda.com/connect/components/processors/protobuf/). - The `dimensions` field to the [`cohere_embeddings` processor](https://docs.redpanda.com/connect/components/processors/cohere_embeddings/). - The `region`, `endpoint`, and `credentials` fields to the `dynamodb` section of the [`aws_kinesis` input](https://docs.redpanda.com/connect/components/inputs/aws_kinesis/). - A `tools` section to the [`cohere_chat` processor](https://docs.redpanda.com/connect/components/processors/cohere_chat/) to enhance response generation with agentic-like behavior. ### [](#deprecations-8)Deprecations This release deprecates the `start_from_oldest` field from the following components: [`kafka_franz` input](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`ockam_kafka` input](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/), [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common` input](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/). Now, you must use the `start_offset` field instead, which defaults to `earliest`. Redpanda recommends upgrading to benefit from this update. ## [](#version-4-52-0)Version 4.52.0 This release introduces three new Slack components: - [`slack_users` input](https://docs.redpanda.com/connect/components/inputs/slack_users/): Reads the full profiles of users in a Slack organization. - [`slack_thread` processor](https://docs.redpanda.com/connect/components/processors/slack_thread/): Reads all messages in a Slack thread. - [`slack_post` output](https://docs.redpanda.com/connect/components/outputs/slack_post/): Posts messages to Slack channel. ## [](#version-4-51-0)Version 4.51.0 Highlights of this update include the introduction of three new components that you could use as part of an AI toolchain: - [`git` input](https://docs.redpanda.com/connect/components/inputs/git/) - [`text_chunker` processor](https://docs.redpanda.com/connect/components/processors/text_chunker/) - [`slack` input](https://docs.redpanda.com/connect/components/inputs/slack/) You can also now [install a FIPS-compliant version of Redpanda Connect](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/#fips-compliance) (for RHEL/Debian Ubuntu). ### [](#component-updates-29)Component updates This release adds: - The `private_key` field to the [`sftp` input](https://docs.redpanda.com/connect/components/inputs/sftp/) and [output](https://docs.redpanda.com/connect/components/outputs/sftp/) - The `history` field to the [`ollama_chat`](https://docs.redpanda.com/connect/components/processors/ollama_chat/) and [`openai_chat_completion`](https://docs.redpanda.com/connect/components/processors/openai_chat_completion/) processors, which allows you to include historical messages in a chat request. - The `handle_logical_types` fields to the [`parquet_decode` processor](https://docs.redpanda.com/connect/components/processors/parquet_decode/). - The `aggregate` operation to the [`mongodb` processor](https://docs.redpanda.com/connect/components/processors/mongodb/), which provides support for aggregation pipelines. - The `headers` option to the `type` field in the [`amqp_0_9` output](https://docs.redpanda.com/connect/components/outputs/amqp_0_9/). ## [](#version-4-50-0)Version 4.50.0 This release introduces agentic behavior to the [`openai_chat_completion` processor](https://docs.redpanda.com/connect/components/processors/openai_chat_completion/). You can now specify external tools to enrich the responses of OpenAI large language models. ### [](#component-updates-30)Component updates This release: - Adds a set of `tools` fields to the [`openai_chat_completion` processor](https://docs.redpanda.com/connect/components/processors/openai_chat_completion/). - Extends support for FLOAT values in the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) to include special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity). Redpanda Connect also now supports the [`unicode_segments` Bloblang method](https://docs.redpanda.com/connect/guides/bloblang/methods/#unicode_segments), which splits a string into segments using [Unicode text segmentation rules](https://hexdocs.pm/unicode_string/readme.html#segmentation). ## [](#version-4-49-1)Version 4.49.1 This release introduces further updates to the [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) and [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). ### [](#component-updates-31)Component updates This release adds: - Two new metrics to the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/): `snowflake_register_latency_ns` and `snowflake_commit_latency_ns`. - An `ignore_nulls` option for schema evolution in the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). - The `translate_ids` field to the [`schema_registry` output](https://docs.redpanda.com/connect/components/outputs/schema_registry/). - The `translate_schema_ids` field to the `redpanda_migrator_bundle` output. Also, the `translate_schema_ids` field in the [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) now defaults to `false`. #### [](#deprecations-9)Deprecations This release deprecates the `snapshot_memory_factor` field from the [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/). Now, you must set a batch size, which defaults to `1000` messages. Redpanda recommends upgrading to benefit from this update. ## [](#version-4-48-1)Version 4.48.1 This release introduces a [change data capture (CDC) input for MongoDB](https://docs.redpanda.com/connect/components/inputs/mongodb_cdc/) on Cloud and Self-Managed Redpanda Connect, along with improvements to the [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) and [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). ### [](#component-updates-32)Component updates This release adds: - The `heartbeat_interval` field to the [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/), which helps manage the size of the Write-Ahead Log when you subscribe to data changes from tables with low activity. - The `commit_timeout` field to the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). - The `url` field to the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/), which allows you to override the default connection URL. - The `is_high_watermark` field to the `redpanda_migrator_offsets` output. - The `kafka_is_high_watermark` metadata field to the `redpanda_migrator_offsets` input. Other changes: - You can now [load an Enterprise Edition license](https://docs.redpanda.com/connect/get-started/licensing/) using the `REDPANDA_LICENSE` environment variable. - The [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) no longer adds the prefix `_rs` to the replication slots it creates. - The `clickhouse` driver is now supported in Redpanda Cloud for all components with the prefix `sql_`. - The [`redpanda migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) now attempts to transfer access control lists for topics that already exist on the destination broker. ## [](#version-4-47-0-to-4-47-1)Version 4.47.0 to 4.47.1 This release introduces: - The [`elasticsearch_v8` output](https://docs.redpanda.com/connect/components/outputs/elasticsearch_v8/): A certified, up-to-date alternative to the existing `elasticsearch` output. - The [`crash` processor](https://docs.redpanda.com/connect/components/processors/crash/): A tool for detecting unhandled pipeline errors during development and testing. ### [](#component-updates-33)Component updates This release adds: - The `arguments` field to the [`amqp_0_9` input](https://docs.redpanda.com/connect/components/inputs/amqp_0_9/) and [output](https://docs.redpanda.com/connect/components/outputs/amqp_0_9/). - The `mapping` field (for Avro messages) to the [`schema_registry_decode` processor](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/). - The `disable_http2` field to the [`http_client` input](https://docs.redpanda.com/connect/components/inputs/http_client/), [output](https://docs.redpanda.com/connect/components/outputs/http_client/), and [`http` processor](https://docs.redpanda.com/connect/components/processors/http/). - The `retry_on_conflict` field to the [`elasticsearch` output](https://docs.redpanda.com/connect/components/outputs/elasticsearch_v8/). Redpanda Connect also now supports the [`uuid_v7` Bloblang function](https://docs.redpanda.com/connect/guides/bloblang/functions/#uuid_v7), which is useful for scenarios when the exact timing of events is important, such as during data migration or replication. ## [](#version-4-46-0)Version 4.46.0 This release introduces a [change data capture (CDC) input for MySQL](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/), and enhancements to the [Bloblang playground](https://docs.redpanda.com/connect/guides/bloblang/playground/), making it easier to test and debug your pipeline configurations. The enhancements include: - A **Run** button in Bloblang code snippets, which opens each snippet in the playground with a pre-filled input and mapping ready for testing. - A **Get help from AI** button, which appears in the playground when an error occurs. ### [](#components-for-cloud-and-self-managed-redpanda-connect-beta)Components for Cloud and Self-Managed Redpanda Connect (beta) - [`mysql_cdc` input](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/) ### [](#component-updates-34)Component updates This release adds: - The `instance_id` field to the [`kafka`](https://docs.redpanda.com/connect/components/inputs/kafka/), [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`ockam_kafka`](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/), [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), and [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) inputs. - The `rebalance_timeout`, `session_timeout`, and `heartbeat_interval` fields to the [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`ockam_kafka`](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/), [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), and [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) inputs. - The `preserve_logical_types` and `raw_unions` fields (for Avro messages), and a `schema_id` metadata field to the [`schema_registry_decode`](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) processor. - The `unchanged_toast_value` field to the [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/). - A `processors` field to the `schema_evolution` object in the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). - New default values to the `max_message_bytes` and `broker_write_max_bytes` fields in the [`kafka_franz`](https://docs.redpanda.com/connect/components/outputs/kafka_franz/), [`ockam_kafka`](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/), [`redpanda`](https://docs.redpanda.com/connect/components/outputs/redpanda/), [`redpanda_migrator`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/), and `redpanda_migrator_offsets` outputs, using IEC instead of SI units. Other changes: - The [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) no longer supports PostgreSQL 10 or 11. #### [](#deprecations-10)Deprecations All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect. | Deprecated fields | Affected components | | --- | --- | | new_column_type_mapping | snowflake_streaming output | ## [](#version-4-45-0)Version 4.45.0 Highlights of this update include the beta release of two components for Self-Managed and Cloud Redpanda Connect, and [a new cookbook](https://docs.redpanda.com/connect/cookbooks/snowflake_ingestion/) that explains how to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. You can also now share links to code snippets in the [Bloblang playground](https://docs.redpanda.com/connect/guides/bloblang/playground/) using the **Share link** button. ### [](#components-for-cloud-and-self-managed-redpanda-connect-beta-2)Components for Cloud and Self-Managed Redpanda Connect (beta) - `ollama_moderation` processor ([Self-Managed](https://docs.redpanda.com/connect/components/processors/ollama_moderation/), [Cloud](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_moderation/)) - `redpanda_migrator_offsets` input (`Self-Managed`, `Cloud`) ### [](#component-updates-35)Component updates This release adds: - The `max_outstanding_messages` and `message_timeout` fields to the [`aws_sqs` input](https://docs.redpanda.com/connect/components/inputs/aws_sqs/). - The `content_type` field to the [`amqp_1` output](https://docs.redpanda.com/connect/components/outputs/amqp_1/). - The `offset_token` field to the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). - The `save_prompt_metadata`, `max_tool_calls` fields and `tools` object to the [`ollama_chat` processor](https://docs.redpanda.com/connect/components/processors/ollama_chat/). - The `offset_topic`, `offset_group`, `offset_partition`, `offset_commit_timestamp` and `offset_metadata` fields to the `redpanda_migrator_offsets` output. - The `topic_lag_refresh_period` field to the [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/) inputs. - The `redpanda_lag` metric to the [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/) inputs. - The `fetch_max_wait` field to the [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`ockam_kafka`](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/), [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), and [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) inputs. - The `kafka_lag` metadata field to the [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/) inputs. - The `avro_schema` and `avro_schema_fingerprint` metadata fields to the [`avro` scanner](https://docs.redpanda.com/connect/components/scanners/avro/). - The `label` field to [template test definitions](https://docs.redpanda.com/connect/configuration/templating/). - The `bloblang` scalar type to [template fields](https://docs.redpanda.com/connect/configuration/templating/). Redpanda Connect now supports: - Interpolation functions for channel and table names in the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). - Calls to external tools when using the [`ollama_chat` processor](https://docs.redpanda.com/connect/components/processors/ollama_chat/). - The execution of multiple database queries as transactions when using the [`sql_raw` input](https://docs.redpanda.com/connect/components/inputs/sql_raw/), [output](https://docs.redpanda.com/connect/components/outputs/sql_raw/) and [processor](https://docs.redpanda.com/connect/components/processors/sql_raw/). - Using the [`label` metadata field](https://docs.redpanda.com/connect/configuration/templating/) within a template mapping to retrieve the template component `label` set in the corresponding configuration. - The following Bloblang functions for error handling: [`error_source_name`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error_source_name), [`error_source_label`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error_source_label), and [`error_source_path`](https://docs.redpanda.com/connect/guides/bloblang/functions/#error_source_path). #### [](#deprecations-11)Deprecations All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect. | Deprecated fields | Affected components | | --- | --- | | batching | redpanda_migrator input, redpanda_migrator output | | kafka_key, max_in_flight | redpanda_migrator_offsets output | | batch_size, multi_header, replication_factor, replication_factor_override, output_resource | redpanda_migrator input | ## [](#version-4-44-0)Version 4.44.0 This release extends the functionality of the [`openai_chat_completion` processor](https://docs.redpanda.com/connect/components/processors/openai_chat_completion/) to include the following fields: `max_tokens`, `temperature`, `user`, `top_p`, `frequency_penalty`, `presence_penalty`, `seed`, and `stop`. ## [](#version-4-43-1)Version 4.43.1 This release introduces support for [trial licenses](https://docs.redpanda.com/connect/get-started/licensing/) for Self-Managed Redpanda Connect. ## [](#version-4-43-0)Version 4.43.0 This release renames the `pg_stream` input to [`postgres_cdc`](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/). The `postgres_cdc` input also no longer emits `mode` metadata, but sets the `operation` metadata field to `read` when a snapshot of a database is processed. ## [](#version-4-42-0)Version 4.42.0 This release introduces a [Bloblang playground](https://docs.redpanda.com/connect/guides/bloblang/playground/) where you can try out your Bloblang syntax and get immediate feedback. It also updates Redpanda Connect licensing checks for Redpanda Connect. Starting from version 4.42.0, you must [add your Enterprise Edition license](https://docs.redpanda.com/connect/get-started/licensing/) to Redpanda Connect before using any Enterprise connectors. ### [](#component-updates-36)Component updates Redpanda Connect now supports: - A Google Cloud `spanner` driver for all SQL plugins. - The following complex data types for the [`pg_stream` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/): JSONB, TEXT\[\], INET, TSVECTOR, TSRANGE, POINT, INTEGER\[\]. - Parquet files for the [`bigquery` output](https://docs.redpanda.com/connect/components/inputs/gcp_bigquery_select/). This release also adds an `exists` operator to the [`cache` processor](https://docs.redpanda.com/connect/components/processors/cache/). ## [](#version-4-41-0)Version 4.41.0 This release adds the `max_records_per_request` field to the [`aws_sqs` output](https://docs.redpanda.com/connect/components/outputs/aws_sqs/). ### [](#packaging-update)Packaging update Starting from version 4.41.0, the `-cgo` suffixed Docker image is no longer available due to low demand and the unacceptable cadence with which the image base (Debian) receives security updates. To create your own CGO builds, run the following command: ```bash CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. ``` ## [](#version-4-40-0)Version 4.40.0 This update includes a beta release of: - The [`pg_stream` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/), which captures data changes made to a PostgreSQL database in real time and streams them to Redpanda Connect. - The [`benchmark` processor](https://docs.redpanda.com/connect/components/processors/benchmark/), which provides message throughput statistics for a data pipeline. It also introduces a new [Bloblang method for timestamps](https://docs.redpanda.com/connect/guides/bloblang/methods/#timestamp). ### [](#components-for-cloud-and-self-managed-redpanda-connect-beta-3)Components for Cloud and Self-Managed Redpanda Connect (beta) - [`pg_stream` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) - [`benchmark` processor](https://docs.redpanda.com/connect/components/processors/benchmark/) ### [](#component-updates-37)Component updates This release adds: - The `metadata_max_age` field to the `redpanda_migrator_offsets` output. - The `timestamp_ms` field to the following outputs: [`ockam_kafka`](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/), [`kafka`](https://docs.redpanda.com/connect/components/outputs/kafka/), [`kafka_franz`](https://docs.redpanda.com/connect/components/outputs/kafka_franz/), [`redpanda`](https://docs.redpanda.com/connect/components/outputs/redpanda/), [`redpanda_common`](https://docs.redpanda.com/connect/components/outputs/redpanda_common/), [`redpanda_migrator`](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/). - The `kafka_timestamp_ms` metadata field, which uses a Unix timestamp with millisecond precision, to the following inputs: [`kafka`](https://docs.redpanda.com/connect/components/inputs/kafka/), [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/), [`redpanda`](https://docs.redpanda.com/connect/components/inputs/redpanda/), [`redpanda_common`](https://docs.redpanda.com/connect/components/inputs/redpanda_common/), [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/). - The `translate_schema_ids` and `schema_registry_output_resource` to the [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/). - The `backfill_dependencies` and `input_resource` fields to the [`schema_registry` output](https://docs.redpanda.com/connect/components/outputs/schema_registry/). - Options to optimize the build of the data output from the [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/). Other changes: - The following components now use the [Franz Kafka Schema Registry client library](https://github.com/twmb/franz-go/tree/master/pkg/sr): [`schema_registry` input](https://docs.redpanda.com/connect/components/inputs/schema_registry/) and [output](https://docs.redpanda.com/connect/components/outputs/schema_registry/), and the [`schema_registry_encode`](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/) and [`schema_registry_decode`](https://docs.redpanda.com/connect/components/processors/schema_registry_decode/) processors. - The [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) can now automatically create tables in Snowflake, when the `schema_evolution.enabled` field is set to `true`. #### [](#deprecations-12)Deprecations **Deprecated field**: `kafka_timestamp` metadata field **Affected components**: - [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/) - [`redpanda_common` input](https://docs.redpanda.com/connect/components/inputs/redpanda_common/) **Replaced by**: `kafka_timestamp_ms` metadata field **Deprecated field**: `timestamp` **Affected components**: - [`ockam_kafka` output](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/) - [`kafka` output](https://docs.redpanda.com/connect/components/outputs/kafka/) - [`kafka_franz` output](https://docs.redpanda.com/connect/components/outputs/kafka_franz/) - [`redpanda` output](https://docs.redpanda.com/connect/components/outputs/redpanda/) - [`redpanda_common` output](https://docs.redpanda.com/connect/components/outputs/redpanda_common/) - [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) **Replaced by**: `timestamp_ms` field **Action required**: Modify all systems that process these fields to ensure compatibility with the new field names. ## [](#version-4-39-0)Version 4.39.0 Highlights of this update include the beta release of three new components for Self-Managed and Cloud, including a new `snowflake_streaming` output. This release also includes additions to the `redpanda` family of components and introduces new features for Self-Managed. ### [](#components-for-cloud-and-self-managed-beta)Components for Cloud and Self-Managed (beta) - [`snowflake_streaming` output](https://docs.redpanda.com/connect/components/outputs/snowflake_streaming/) - [`timeplus` input](https://docs.redpanda.com/connect/components/inputs/timeplus/) - [`spicedb_watch` input](https://docs.redpanda.com/connect/components/inputs/spicedb_watch/) ### [](#additional-redpanda-components-beta)Additional `redpanda` components (beta) - `redpanda` ([input](https://docs.redpanda.com/connect/components/inputs/redpanda/) and [output](https://docs.redpanda.com/connect/components/outputs/redpanda/)): Consume message data from or send message data to Kafka brokers. - `redpanda_common` ([input](https://docs.redpanda.com/connect/components/inputs/redpanda_common/) and [output](https://docs.redpanda.com/connect/components/outputs/redpanda_common/)): Consume message data from or send message data to a Redpanda Kafka broker, using credentials from a common [`redpanda` configuration block](https://docs.redpanda.com/connect/components/redpanda/about/) (Self-Managed only). ### [](#features-in-self-managed)Features in Self-Managed You can now: - Create an [allow list or deny list](https://docs.redpanda.com/connect/configuration/allow_and_deny_lists/) for a Redpanda Connect instance - Use the [`rpk connect` CLI flag `--secrets`](https://docs.redpanda.com/connect/configuration/secrets/#look-up-secrets-on-a-remote-system-at-runtime) to look up secrets on a remote system - Use the [`rpk connect` CLI flag `--disable-telemetry`](https://docs.redpanda.com/connect/configuration/telemetry/#disable-the-telemetry-service) to block the telemetry service ### [](#component-updates-38)Component updates This release adds: - The `follow_redirects` field to the [`http` processor](https://docs.redpanda.com/connect/components/processors/http/). - The following fields to the [`redpanda` configuration service](https://docs.redpanda.com/connect/components/redpanda/about/): - `metadata_max_age` - `partitioner` - `idempotent_write` - `compression` - `broker_write_max_bytes` - The `metadata_max_age` field to the [`kafka_franz` output](https://docs.redpanda.com/connect/components/outputs/kafka_franz/). - The following fields to the [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/) and [`ockam_kafka`](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/) inputs: - `fetch_max_bytes` - `fetch_min_bytes` - `fetch_max_partition_bytes` - The `url` field of the [`aws_sqs` output](https://docs.redpanda.com/connect/components/outputs/aws_sqs/) now supports interpolation functions. #### [](#deprecations-13)Deprecations All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect. | Deprecated fields | Affected components | | --- | --- | | auto_replay_nacks, sasl, metadata_max_age | ockam_kafka input | | client_id, rack_id, sasl | ockam_kafka output | | rack_id | kafka_franz output | ## [](#version-4-38-0)Version 4.38.0 This update includes the beta release of two new outputs for Cloud and Self-Managed, and the introduction of a [telemetry service](https://docs.redpanda.com/connect/configuration/telemetry/), which collects and sends anonymized usage statistics to Redpanda. ### [](#new-tooling)New tooling A new Helm chart for deploying Redpanda Connect in Kubernetes is now available. This chart simplifies the deployment and configuration of Redpanda Connect, allowing you to manage and scale Redpanda Connect instances in Kubernetes environments. To get started with the Helm chart, see [Get Started with the Redpanda Connect Helm Chart](https://docs.redpanda.com/connect/get-started/quickstarts/helm-chart/). Or, see the release notes for the chart on [GitHub](https://github.com/redpanda-data/helm-charts/releases/tag/connect-3.0.0). ### [](#new-outputs-beta)New outputs (beta) - [`azure_data_lake_gen2`](https://docs.redpanda.com/connect/components/outputs/azure_data_lake_gen2/) (certified) - [`timeplus`](https://docs.redpanda.com/connect/components/outputs/timeplus/) (community) ### [](#component-updates-39)Component updates This release adds: - The `checksum_algorithm` to the [`aws_s3` output](https://docs.redpanda.com/connect/components/outputs/aws_s3/). - The `private_key` field to the [`snowflake_put` output](https://docs.redpanda.com/connect/components/outputs/snowflake_put/). - The `nkey` field to the following components: - `nats` ([input](https://docs.redpanda.com/connect/components/inputs/nats/), [output](https://docs.redpanda.com/connect/components/outputs/nats/)) - `nats_jetstream` ([input](https://docs.redpanda.com/connect/components/inputs/nats_jetstream/), [output](https://docs.redpanda.com/connect/components/outputs/nats_jetstream/)) - `nats_kv` ([input](https://docs.redpanda.com/connect/components/inputs/nats_kv/), [output](https://docs.redpanda.com/connect/components/outputs/nats_kv/), [processor](https://docs.redpanda.com/connect/components/processors/nats_kv/)) - `nats_stream` ([input](https://docs.redpanda.com/connect/components/inputs/nats_stream/), [output](https://docs.redpanda.com/connect/components/outputs/nats_stream/)) - `nats_request_reply` ([processor](https://docs.redpanda.com/connect/components/processors/nats_request_reply/)) Other changes: - The `collection` field of the [`mongodb` output](https://docs.redpanda.com/connect/components/outputs/mongodb/) now supports interpolation functions. - All components that used the deprecated `count` function, with a default value for the `path` field, now use the `counter` function. > 📝 **NOTE** > > If you have multiple components that execute a mapping using the deprecated `count` function with the original default value, and these counters need to cascade, there may be a change in behavior. To avoid this, explicitly define a value for the `path` field. ## [](#version-4-37-0)Version 4.37.0 Redpanda Connect has new bundled [Redpanda Migrator components](#redpanda-migrator-components-beta), more [AI processors](#ai-processors-beta) for Self-Managed and Cloud platforms, and three new [outputs](#certified-and-community-outputs-beta). ### [](#redpanda-migrator-components-beta)Redpanda Migrator components (beta) You can now move workloads from any Apache Kafka system to Redpanda using the `redpanda_migrator_bundle` input and `output`. For full details, see the Redpanda Migrator Cookbook for [Self-Managed](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/) and [Cloud](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/redpanda_migrator/). ### [](#ai-processors-beta)AI processors (beta) Some new additions to Redpanda Connect’s growing list of AI processors: - [`aws_bedrock_embeddings`](https://docs.redpanda.com/connect/components/processors/aws_bedrock_embeddings/) - [`cohere_chat`](https://docs.redpanda.com/connect/components/processors/cohere_chat/) - [`cohere_embeddings`](https://docs.redpanda.com/connect/components/processors/cohere_embeddings/) - [`gcp_vertex_ai_embeddings`](https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_embeddings/) ### [](#certified-and-community-outputs-beta)Certified and community outputs (beta) - [`couchbase`](https://docs.redpanda.com/connect/components/outputs/couchbase/) (community) - [`cypher`](https://docs.redpanda.com/connect/components/outputs/cypher/) (community) - [`questdb`](https://docs.redpanda.com/connect/components/outputs/questdb/) (certified) ### [](#component-updates-40)Component updates This release adds: - The `metadata_max_age` field to [`kafka_franz`](https://docs.redpanda.com/connect/components/inputs/kafka_franz/) and [`redpanda_migrator`](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) inputs. - The `fetch_in_order` field to the [`schema_registry` input](https://docs.redpanda.com/connect/components/inputs/schema_registry/). - The `encrypted_fields` fields to the `ockam_kafka` [input](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/) and [output](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/). ## [](#version-4-36-0)Version 4.36.0 You can now use the `replication_factor_override` and `replication_factor` fields of the [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) and [output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) to set a replication factor for migrated topics. ## [](#versions-4-35-1-to-4-35-3)Versions 4.35.1 to 4.35.3 Redpanda Connect is now fully integrated with the Redpanda Cloud platform. This release adds Azure and GCP components to the growing list of available components. For a full list of components supported in Redpanda Cloud, see the [Connector Catalog](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/). ## [](#version-4-35-0)Version 4.35.0 Redpanda Connect now includes beta versions of Redpanda Migrator components: - [`redpanda_migrator` input](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) and [output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) - `redpanda_migrator_bundle` input and `output` - `redpanda_migrator_offsets` output For full details, see the Redpanda Migrator Cookbook for [Self-Managed](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/) and [Cloud](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/redpanda_migrator/). ### [](#component-updates-41)Component updates This release adds: - OAuth, basic authentication, and JSON Web Token fields to the [`schema_registry` input](https://docs.redpanda.com/connect/components/inputs/schema_registry/) and [output](https://docs.redpanda.com/connect/components/outputs/schema_registry/). - The `job_project` field to the [`gcp_bigquery` output](https://docs.redpanda.com/connect/components/outputs/gcp_bigquery/). - The `broker_write_max_bytes` field to the [`kafka_franz`](https://docs.redpanda.com/connect/components/outputs/kafka_franz/) and [`ockam_kafka`](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/) outputs. ## [](#version-4-34-0)Version 4.34.0 Redpanda Connect now includes beta versions of two new AI processors for Cloud and Self-Managed platforms: - [`aws_bedrock_chat`](https://docs.redpanda.com/connect/components/processors/aws_bedrock_chat/) - [`gcp_vertex_ai_chat`](https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_chat/) ## [](#version-4-33-0)Version 4.33.0 Redpanda Connect now includes beta versions of four new components. ### [](#for-self-managed-and-cloud-platforms)For Self-Managed and Cloud platforms - [`schema_registry` input](https://docs.redpanda.com/connect/components/inputs/schema_registry/) and [output](https://docs.redpanda.com/connect/components/outputs/schema_registry/) - [`qdrant` output](https://docs.redpanda.com/connect/components/outputs/qdrant/) ### [](#for-self-managed-only)For Self-Managed only - [`ockam_kafka` input](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/) and [output](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/) You can also use the new Bloblang [`vector`](https://docs.redpanda.com/connect/guides/bloblang/methods/) method to create vector embeddings for SQL databases. ### [](#component-updates-42)Component updates This release adds: - The `credentials_json` field to all GCP components. - The `content_md5` field to the [`aws_s3` output](https://docs.redpanda.com/connect/components/outputs/aws_s3/). - The `send_ack` field to the [`nats` input](https://docs.redpanda.com/connect/components/inputs/nats/). ## [](#version-4-32-1)Version 4.32.1 Redpanda Connect has a new batch of beta AI processors for Self-Managed and Cloud platforms. ### [](#for-self-managed-and-cloud-platforms-2)For Self-Managed and Cloud platforms - [`openai_chat_completion`](https://docs.redpanda.com/connect/components/processors/openai_chat_completion/) - [`openai_embeddings`](https://docs.redpanda.com/connect/components/processors/openai_embeddings/) - [`openai_image_generation`](https://docs.redpanda.com/connect/components/processors/openai_image_generation/) - [`openai_speech`](https://docs.redpanda.com/connect/components/processors/openai_speech/) - [`openai_transcription`](https://docs.redpanda.com/connect/components/processors/openai_transcription/) - [`openai_translation`](https://docs.redpanda.com/connect/components/processors/openai_translation/) ### [](#for-self-managed-only-2)For Self-Managed only - [`ollama_chat`](https://docs.redpanda.com/connect/components/processors/ollama_chat/) - [`ollama_embeddings`](https://docs.redpanda.com/connect/components/processors/ollama_embeddings/) ## [](#version-4-31-0)Version 4.31.0 Redpanda Connect now includes beta versions of two new components: - [`pinecone` output](https://docs.redpanda.com/connect/components/outputs/pinecone/) - Redpanda Connect’s first AI component - [`redpanda_data_transform` processor](https://docs.redpanda.com/connect/components/processors/redpanda_data_transform/) ### [](#component-updates-43)Component updates This release adds: - Custom TLS fields for the [`splunk` input](https://docs.redpanda.com/connect/components/inputs/splunk/) and [`splunk_hec` output](https://docs.redpanda.com/connect/components/outputs/splunk_hec/). - The `timestamp` field to [`kafka`](https://docs.redpanda.com/connect/components/outputs/kafka/) and [`kafka_franz`](https://docs.redpanda.com/connect/components/outputs/kafka_franz/) outputs. - The `status_topic` field to the [`redpanda` component](https://docs.redpanda.com/connect/components/redpanda/about/). ## [](#version-4-30-0-to-4-30-1)Version 4.30.0 to 4.30.1 Redpanda Connect now includes a beta version of the [`splunk` input](https://docs.redpanda.com/connect/components/inputs/splunk/). ## [](#version-4-29-0)Version 4.29.0 Redpanda Connect now separates packages into an enterprise bundle and a free bundle, which contains community and selected certified connectors. To learn more about licensing and support levels, see [components:about.adoc#support-levels](https://docs.redpanda.com/connect/components/about/#support-levels). ### [](#component-updates-44)Component updates This release adds: - The `auth.oauth2.scope` field to the [`pulsar` input](https://docs.redpanda.com/connect/components/inputs/pulsar/) and [output](https://docs.redpanda.com/connect/components/outputs/pulsar/). - The `subscription_initial_position` field to the [`pulsar` input](https://docs.redpanda.com/connect/components/inputs/pulsar/). --- # Page 375: Guides **URL**: https://docs.redpanda.com/connect/guides.md --- # Guides --- title: Guides latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/index.adoc description: Explore guides for installing, configuring, and running Redpanda Connect. page-git-created-date: "2024-06-03" page-git-modified-date: "2026-08-12" --- - [Monitoring](monitoring/) Learn how to monitor Redpanda Connect, including setting up metrics and alerts to ensure system health and performance. - [Monitor Pipeline Status](monitor-pipeline-status/) Monitor Redpanda Connect pipeline health using status events, set up alerts for connection failures, and troubleshoot common issues. - [Message Delivery Semantics](delivery_semantics/) Learn about Redpanda Connect's transactional model and error handling - [Performance Tuning](performance_tuning/) Discover best practices for tuning Redpanda Connect performance, optimizing configurations, and achieving maximum throughput. - [Tune the Iceberg output for throughput](iceberg-output-tuning/) Configure the Redpanda Connect Iceberg output for high write throughput using tested recipes for append-only and keyed CDC workloads. - [Synchronous Responses](sync_responses/) Understand synchronous response handling in Redpanda Connect, ensuring reliable and efficient data processing. - [Change Data Capture (CDC)](cdc/) Capture database changes in real time and stream them to Redpanda topics using native CDC inputs. - [Cloud Credentials](cloud/) Explore how to deploy and manage Redpanda Connect in the cloud, including setup, configuration, and optimization tips. - [Serverless](serverless/about/) Learn about serverless architecture in Redpanda Connect, including benefits, use cases, and implementation details. - [Streams Mode](streams_mode/about/) Get an overview of streams mode in Redpanda Connect, detailing its features, use cases, and setup instructions. - [Migration](migration/) Migrate between different Redpanda Connect versions, including preparation steps and best practices. --- # Page 376: Bloblang **URL**: https://docs.redpanda.com/connect/guides/bloblang/about.md --- # Bloblang --- title: Bloblang latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/about.adoc description: Learn what Bloblang is and how to use the native mapping language. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-13" --- Bloblang, or blobl for short, is a language designed for mapping data of a wide variety of forms. It’s a safe, fast, and powerful way to perform document mapping within Redpanda Connect. It also has a [Go API for writing your own functions and methods](https://pkg.go.dev/github.com/redpanda-data/connect/v4/public/bloblang) as plugins. Bloblang is available as a [processor](https://docs.redpanda.com/connect/components/processors/mapping/) and it’s also possible to use blobl queries in [function interpolations](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). You can also execute Bloblang mappings on the command-line with the `blobl` subcommand: ```shell cat data.jsonl | rpk connect blobl 'foo.(bar | baz).buz' ``` This document outlines the core features of the Bloblang language, but if you’re totally new to Bloblang then it’s worth following [the walkthrough first](https://docs.redpanda.com/connect/guides/bloblang/walkthrough/). ## [](#learn-bloblang)Learn Bloblang [learnbloblang.com](https://www.learnbloblang.com) is an interactive resource for learning Bloblang with hands-on exercises. ## [](#assignment)Assignment A Bloblang mapping expresses how to create a new document by extracting data from an existing input document. Assignments consist of a dot separated path segments on the left-hand side describing a field to be created within the new document, and a right-hand side query describing what the content of the new field should be. The keyword `root` on the left-hand side refers to the root of the new document, the keyword `this` on the right-hand side refers to the current context of the query, which is the read-only input document when querying from the root of a mapping: ```bloblang root.id = this.thing.id root.type = "yo" # Both `root` and `this` are optional, and will be inferred in their absence. content = thing.doc.message # In: {"thing":{"id":"wat1","doc":{"title":"wut","message":"hello world"}}} ``` Since the document being created starts off empty it is sometimes useful to begin a mapping by copying the entire contents of the input document, which can be expressed by assigning `this` to `root`. ```bloblang root = this root.foo = "added value" # In: {"id":"wat1","message":"hello world"} ``` If the new document `root` is never assigned to or otherwise mutated then the original document remains unchanged. ### [](#special-characters-in-paths)Special characters in paths Quotes can be used to describe sections of a field path that contain whitespace, dots or other special characters: ```bloblang # Use quotes around a path segment in order to include whitespace or dots within # the path root."foo.bar".baz = this."buz bev".fub # In: {"buz bev":{"fub":"hello world"}} ``` ### [](#non-structured-data)Non-structured data Bloblang is able to map data that is unstructured, whether it’s a log line or a binary blob, by referencing it with the [`content` function](https://docs.redpanda.com/connect/guides/bloblang/functions/#content), which returns the raw bytes of the input document: ```bloblang # Parse a base64 encoded JSON document root = content().decode("base64").parse_json() # In: eyJmb28iOiJiYXIifQ== ``` And your newly mapped document can also be unstructured, simply assign a value type to the `root` of your document: ```bloblang root = this.foo # In: {"foo":"hello world"} ``` And the resulting message payload will be the raw value you’ve assigned. ### [](#deleting)Deleting It’s possible to selectively delete fields from an object by assigning the function `deleted()` to the field path: ```bloblang root = this root.bar = deleted() # In: {"id":"wat1","message":"hello world","bar":"remove me"} ``` ### [](#variables)Variables Another type of assignment is a `let` statement, which creates a variable that can be referenced elsewhere within a mapping. Variables are discarded at the end of the mapping and are mostly useful for query reuse. Variables are referenced within queries with `$`: ```bloblang # Set a temporary variable let foo = "yo" root.new_doc.type = $foo ``` ### [](#metadata)Metadata Redpanda Connect messages contain metadata that is separate from the main payload, in Bloblang you can modify the metadata of the resulting message with the `meta` assignment keyword. Metadata values of the resulting message are referenced within queries with the `@` operator or the [`metadata()` function](https://docs.redpanda.com/connect/guides/bloblang/functions/#metadata): ```bloblang # Reference a metadata value root.new_doc.bar = @kafka_topic # Or `@.kafka_topic` or `metadata("kafka_topic")` # Delete all metadata meta = deleted() # Set metadata values meta bar = "hello world" meta baz = { "something": "structured" } # Get an object of key/values for all metadata root.meta_obj = @ # Or `metadata()` ``` ## [](#coalesce)Coalesce The pipe operator (`|`) used within brackets allows you to coalesce multiple candidates for a path segment. The first field that exists and has a non-null value will be selected: ```bloblang root.new_doc.type = this.thing.(article | comment | this).type # In: {"thing":{"article":{"type":"foo"}}} # In: {"thing":{"comment":{"type":"bar"}}} # In: {"thing":{"type":"baz"}} ``` Opening brackets on a field begins a query where the context of `this` changes to value of the path it is opened upon, therefore in the above example `this` within the brackets refers to the contents of `this.thing`. ## [](#literals)Literals Bloblang supports number, boolean, string, null, array and object literals: ```bloblang root = [ 7, false, "string", null, { "first": 11, "second": {"foo":"bar"}, "third": """multiple lines on this string""" } ] # In: {} ``` The values within literal arrays and objects can be dynamic query expressions, as well as the keys of object literals. ## [](#comments)Comments You might’ve already spotted, comments are started with a hash (`#`) and end with a line break: ```bloblang root = this.some.value # And now this is a comment ``` ## [](#boolean-logic-and-arithmetic)Boolean logic and arithmetic Bloblang supports a range of boolean operators `!`, `>`, `>=`, `==`, `<`, `<=`, `&&`, `||` and mathematical operators `+`, `-`, `*`, `/`, `%`: ```bloblang root.is_big = this.number > 100 root.multiplied = this.number * 7 # In: {"number":50} # In: {"number":150} ``` For more information about these operators and how they work check out [the arithmetic page](https://docs.redpanda.com/connect/guides/bloblang/arithmetic/). ## [](#conditional-mapping)Conditional mapping Use `if` as either a statement or an expression in order to perform maps conditionally: ```bloblang root = this root.sorted_foo = if this.foo.type() == "array" { this.foo.sort() } if this.foo.type() == "string" { root.upper_foo = this.foo.uppercase() root.lower_foo = this.foo.lowercase() } # In: {"foo":"FooBar"} # In: {"foo":["foo","bar"]} ``` And add as many `else if` queries as you like, followed by an optional final fallback `else`: ```bloblang root.sound = if this.type == "cat" { this.cat.meow } else if this.type == "dog" { this.dog.woof.uppercase() } else { "sweet sweet silence" } # In: {"type":"cat","cat":{"meow":"meeeeooooow!"}} # In: {"type":"dog","dog":{"woof":"guurrrr woof woof!"}} # In: {"type":"caterpillar","caterpillar":{"name":"oleg"}} ``` ## [](#pattern-matching)Pattern matching A `match` expression allows you to perform conditional mappings on a value, each case should be either a boolean expression, a literal value to compare against the target value, or an underscore (`_`) which captures values that have not matched a prior case: ```bloblang root.new_doc = match this.doc { this.type == "article" => this.article this.type == "comment" => this.comment _ => this } # In: {"doc":{"type":"article","article":{"id":"foo","content":"qux"}}} # In: {"doc":{"type":"comment","comment":{"id":"bar","content":"quz"}}} # In: {"doc":{"type":"neither","content":"some other stuff unchanged"}} ``` Within a match block the context of `this` changes to the pattern matched expression, therefore `this` within the match expression above refers to `this.doc`. Match cases can specify a literal value for simple comparison: ```bloblang root = this root.type = match this.type { "doc" => "document", "art" => "article", _ => this } # In: {"type":"doc","foo":"bar"} ``` The match expression can also be left unset which means the context remains unchanged, and the catch-all case can also be omitted: ```bloblang root.new_doc = match { this.doc.type == "article" => this.doc.article this.doc.type == "comment" => this.doc.comment } # In: {"doc":{"type":"neither","content":"some other stuff unchanged"}} ``` If no case matches then the mapping is skipped entirely, hence we would end up with the original document in this case. ## [](#functions)Functions Functions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped: ```bloblang root.doc.id = uuid_v4() root.doc.received_at = now() root.doc.host = hostname() ``` Functions support both named and nameless style arguments: ```bloblang root.values_one = range(start: 0, stop: this.max, step: 2) root.values_two = range(0, this.max, 2) # In: {"max":10} ``` You can find a full list of functions and their parameters in [the functions page](https://docs.redpanda.com/connect/guides/bloblang/functions/). ## [](#methods)Methods Methods are similar to functions but enact upon a target value, these provide most of the power in Bloblang as they allow you to augment query values and can be added to any expression (including other methods): ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,8,15,20]},"user":{"fav_pokemon":"pikachu"}} ``` Methods also support both named and nameless style arguments: ```bloblang root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} ``` You can find a full list of methods and their parameters in [the methods page](https://docs.redpanda.com/connect/guides/bloblang/methods/). ## [](#maps)Maps Defining named maps allows you to reuse common mappings on values with the [`apply` method](https://docs.redpanda.com/connect/guides/bloblang/methods/#apply): ```bloblang map things { root.first = this.thing_one root.second = this.thing_two } root.foo = this.value_one.apply("things") root.bar = this.value_two.apply("things") # In: {"value_one":{"thing_one":"hey","thing_two":"yo"},"value_two":{"thing_one":"sup","thing_two":"waddup"}} ``` Within a map the keyword `root` refers to a newly created document that will replace the target of the map, and `this` refers to the original value of the target. The argument of `apply` is a string, which allows you to dynamically resolve the mapping to apply. ## [](#import-maps)Import maps It’s possible to import maps defined in a file with an `import` statement: ```bloblang import "./common_maps.blobl" root.foo = this.value_one.apply("things") root.bar = this.value_two.apply("things") # In: {"value_one":{"thing_one":"hey","thing_two":"yo"},"value_two":{"thing_one":"sup","thing_two":"waddup"}} ``` Imports from a Bloblang mapping within a Redpanda Connect config are relative to the process running the config. Imports from an imported file are relative to the file that is importing it. ## [](#filtering)Filtering By assigning the root of a mapped document to the `deleted()` function you can delete a message entirely: ```bloblang # Filter all messages that have fewer than 10 URLs. root = if this.doc.urls.length() < 10 { deleted() } # In: {"doc":{"urls":["a","b","c"]}} # In: {"doc":{"urls":["a","b","c","d","e","f","g","h","i","j"]}} ``` ## [](#error-handling)Error handling Functions and methods can fail under certain circumstances, such as when they receive types they aren’t able to act upon. These failures, when not caught, will cause the entire mapping to fail. However, the [method `catch`](https://docs.redpanda.com/connect/guides/bloblang/methods/#catch) can be used in order to return a value when a failure occurs instead: ```bloblang # Map an empty array to `foo` if the field `bar` is not a string. root.foo = this.bar.split(",").catch([]) # In: {"bar":"a,b,c"} # In: {"bar":123} ``` Since `catch` is a method it can also be attached to bracketed map expressions: ```bloblang # Map `false` if any of the operations in this boolean query fail. root.thing = ( this.foo > this.bar && this.baz.contains("wut") ).catch(false) # In: {"foo":10,"bar":5,"baz":"wut wut"} # In: {"foo":"not a number","bar":5,"baz":"wut wut"} ``` And one of the more powerful features of Bloblang is that a single `catch` method at the end of a chain of methods can recover errors from any method in the chain: ```bloblang # Catch errors caused by: # - foo not existing # - foo not being a string # - an element from split foo not being a valid JSON string root.things = this.foo.split(",").map_each( ele -> ele.parse_json() ).catch([]) # Specifically catch a JSON parse error root.things = this.foo.split(",").map_each( ele -> ele.parse_json().catch({}) ) # In: {"foo":"{\"a\":1},{\"b\":2}"} # In: {"foo":"not valid json"} ``` However, the `catch` method only acts on errors, sometimes it’s also useful to set a fall back value when a query returns `null` in which case the [method `or`](https://docs.redpanda.com/connect/guides/bloblang/methods/#or) can be used the same way: ```bloblang # Map "default" if either the element index 5 does not exist, or the underlying # element is `null`. root.foo = this.bar.index(5).or("default") # In: {"bar":["a","b","c"]} # In: {"bar":["a","b","c","d","e","f","g"]} ``` ## [](#unit-testing)Unit testing It’s possible to execute unit tests for your Bloblang mappings using the standard Redpanda Connect unit test capabilities outlined [in this document](https://docs.redpanda.com/connect/configuration/unit_testing/). ## [](#troubleshooting)Troubleshooting 1. I’m seeing `unable to reference message as structured (with 'this')` when I try to run mappings with `rpk connect blobl`. That particular error message means the mapping is failing to parse what’s being fed in as a JSON document. Make sure that the data you are feeding in is valid JSON, and also that the documents _do not_ contain line breaks as `rpk connect blobl` will parse each line individually. Why? That’s a good question. Bloblang supports non-JSON formats too, so it can’t delimit documents with a streaming JSON parser like tools such as `jq`, so instead it uses line breaks to determine the boundaries of each message. --- # Page 377: Advanced Bloblang **URL**: https://docs.redpanda.com/connect/guides/bloblang/advanced.md --- # Advanced Bloblang --- title: Advanced Bloblang latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/advanced page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/advanced.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/advanced.adoc description: Some advanced Bloblang patterns page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-04" --- ## [](#map-parameters)Map parameters A map definition only has one input parameter, which is the context that it is called upon: ```bloblang map formatting { root = "(%v)".format(this) } root.a = this.a.apply("formatting") root.b = this.b.apply("formatting") # In: {"a":"foo","b":"bar"} ``` However, we can use object literals in order to provide multiple map parameters. Imagine if we wanted a map that is the exact same as above except the pattern is `[%v]` instead, with the potential for even more patterns in the future. To do that we can pass an object with a field `value` with our target to map and a field `pattern` which allows us to specify the pattern to apply: ```bloblang map formatting { root = this.pattern.format(this.value) } root.a = { "value":this.a, "pattern":this.pattern, }.apply("formatting") root.b = { "value":this.b, "pattern":this.pattern, }.apply("formatting") # In: {"a":"foo","b":"bar","pattern":"[%v]"} ``` ## [](#walking-the-tree)Walking the tree Sometimes it’s necessary to perform a mapping on all values within an unknown tree structure. You can do that easily with recursive mapping: ```bloblang map unescape_values { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("unescape_values")), this.type() == "array" => this.map_each(ele -> ele.apply("unescape_values")), this.type() == "string" => this.unescape_html(), this.type() == "bytes" => this.unescape_html(), _ => this, } } root = this.apply("unescape_values") # In: {"first":{"nested":"foo & bar"},"second":10,"third":["1 < 2",{"also_nested":"2 > 1"}]} ``` ## [](#message-expansion)Message expansion Expanding a single message into multiple messages can be done by mapping messages into an array and following it up with an [`unarchive` processor](https://docs.redpanda.com/connect/components/processors/unarchive/). For example, given documents of this format: ```json { "id": "foobar", "items": [ {"content":"foo"}, {"content":"bar"}, {"content":"baz"} ] } ``` We can pull `items` out to the root with `root = items` with a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) and follow it with an [`unarchive` processor](https://docs.redpanda.com/connect/components/processors/unarchive/) to expand each element into its own independent message: ```yaml pipeline: processors: - mapping: root = this.items - unarchive: format: json_array ``` However, most of the time we also need to map the elements before expanding them, and often that includes copying fields outside of our target array. We can do that with methods such as `map_each` and `merge`: ```bloblang # In: {"id":"foobar","items":[{"content":"foo"},{"content":"bar"},{"content":"baz"}]} root = this.items.map_each(ele -> this.without("items").merge(ele)) ``` However, the above mapping is slightly inefficient as we would create a copy of our source object for each element with the `this.without("items")` part. A more efficient way to do this would be to capture that query within a variable: ```bloblang # In: {"id":"foobar","items":[{"content":"foo"},{"content":"bar"},{"content":"baz"}]} let doc_root = this.without("items") root = this.items.map_each($doc_root.merge(this)) ``` Also note that when we set `doc_root` we remove the field `items` from the target document. The full config would now be: ```yaml pipeline: processors: - mapping: | let doc_root = this.without("items") root = this.items.map_each($doc_root.merge(this)) - unarchive: format: json_array ``` ## [](#creating-csv)Creating CSV Redpanda Connect has a few different ways of outputting a stream of CSV data. However, the best way to do it is by converting the documents into CSV rows with Bloblang as this gives you full control over exactly how the schema is generated, erroneous data is handled, and escaping of column data is performed. A common and simple use case is to simply flatten documents and write out the column values in alphabetical order. The first row we generate should also be prefixed with a row containing those column names. Here’s a mapping that achieves this by using a `count` function to detect the very first invocation of the mapping in a stream pipeline: ```bloblang # In: {"name":"foo","address":"123 Main St, Apt 2"} # foo,"123 Main St, Apt 2" map escape_csv { root = if this.re_match("[\"\n,]+") { "\"" + this.replace_all("\"", "\"\"") + "\"" } else { this } } # Extract key/value pairs as an array and sort by the key let kvs = this.key_values().sort_by(v -> v.key) # Create a header prefix for our output only on the first row let header = if count("rows_in_file") == 1 { $kvs.map_each(kv -> kv.key.apply("escape_csv")).join(",") + "\n" } else { "" } root = $header + $kvs.map_each(kv -> kv.value.string().apply("escape_csv")).join(",") ``` And with this mapping we can write the data to a newly created CSV file using an output with a simple `lines` codec: ```yaml output: file: path: ./result.csv codec: lines ``` Perhaps the first expansion of this mapping that would be worthwhile is to add an explicit list of column names, or at least confirm that the number of values in a row matches an expected count. --- # Page 378: Bloblang Arithmetic **URL**: https://docs.redpanda.com/connect/guides/bloblang/arithmetic.md --- # Bloblang Arithmetic --- title: Bloblang Arithmetic latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/arithmetic page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/arithmetic.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/arithmetic.adoc description: How arithmetic works within Bloblang page-git-created-date: "2024-05-24" page-git-modified-date: "2024-11-25" --- Bloblang supports a range of comparison operators `!`, `>`, `>=`, `==`, `<`, `<=`, `&&`, `||` and mathematical operators `+`, `-`, `*`, `/`, `%`. How these operators behave is dependent on the type of the values they’re used with, and therefore it’s worth fully understanding these behaviors if you intend to use them heavily in your mappings. ## [](#mathematical)Mathematical All mathematical operators (`+`, `-`, `*`, `/`, `%`) are valid against number values, and addition (`+`) is also supported when both the left and right hand side arguments are strings. If a mathematical operator is used with an argument that is non-numeric (with the aforementioned string exception) then a [recoverable mapping error will be thrown](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). ### [](#number-degradation)Number degradation In Bloblang any number resulting from a method, function or arithmetic is either a 64-bit signed integer or a 64-bit floating point value. Numbers from input documents can be any combination of size and be signed or unsigned. When a mathematical operation is performed with two or more integer values Bloblang will create an integer result, with the exception of division. However, if any number within a mathematical operation is a floating point then the result will be a floating point value. In order to explicitly coerce numbers into integer types you can use the [`.ceil()`, `.floor()`, or `.round()` methods](https://docs.redpanda.com/connect/guides/bloblang/methods/#number-manipulation). ## [](#comparison)Comparison The not (`!`) operator reverses the boolean value of the expression immediately following it, and is valid to place before any query that yields a boolean value. If the following expression yields a non-boolean value then a [recoverable mapping error will be thrown](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). If you wish to reverse the boolean result of a complex query then simply place the query within brackets (`!(this.foo > this.bar)`). ### [](#equality)Equality The equality operators (`==` and `!=`) are valid to use against any value type. In order for arguments to be considered equal they must match in both their basic type (`string`, `number`, `null`, `bool`, etc) as well as their value. If you wish to compare mismatched value types then use [coercion methods](https://docs.redpanda.com/connect/guides/bloblang/methods/#type-coercion). Number arguments are considered equal if their value is the same when represented the same way, which means their underlying representations (integer, float, etc) do not need to match in order for them to be considered equal. ### [](#numerical)Numerical Numerical comparisons (`>`, `>=`, `<`, `<=`) are valid to use against number values only. If a non-number value is used as an argument then a [recoverable mapping error will be thrown](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). ### [](#boolean)Boolean Boolean comparison operators (`||`, `&&`) are valid to use against boolean values only (`true` or `false`). If a non-boolean value is used as an argument then a [recoverable mapping error will be thrown](https://docs.redpanda.com/connect/guides/bloblang/about/#error-handling). --- # Page 379: Bloblang Functions **URL**: https://docs.redpanda.com/connect/guides/bloblang/functions.md --- # Bloblang Functions --- title: Bloblang Functions latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/functions page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/functions.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/functions.adoc description: A list of Bloblang functions page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Functions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped: ```bloblang root.doc.id = uuid_v4() root.doc.received_at = now() root.doc.host = hostname() ``` Functions support both named and nameless style arguments: ```bloblang root.values_one = range(start: 0, stop: this.max, step: 2) root.values_two = range(0, this.max, 2) # In: {"max":10} ``` ## [](#batch_index)batch_index Returns the zero-based index of the current message within its batch. Use this to conditionally process messages based on their position, or to create sequential identifiers within a batch. ### [](#examples)Examples ```bloblang root = if batch_index() > 0 { deleted() } ``` Create a unique identifier combining batch position with timestamp: ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` ## [](#batch_size)batch_size Returns the total number of messages in the current batch. Use this to determine batch boundaries or compute relative positions. ### [](#examples-2)Examples ```bloblang root.total = batch_size() ``` Check if processing the last message in a batch: ```bloblang root.is_last = batch_index() == batch_size() - 1 ``` ## [](#bytes)bytes Creates a zero-initialized byte array of specified length. Use this to allocate fixed-size byte buffers for binary data manipulation or to generate padding. ### [](#parameters)Parameters | Name | Type | Description | | --- | --- | --- | | length | integer | The size of the resulting byte array. | ### [](#examples-3)Examples ```bloblang root.data = bytes(5) ``` Create a buffer for binary operations: ```bloblang root.header = bytes(16) root.payload = content() ``` ## [](#content)content Returns the raw message payload as bytes, regardless of the current mapping context. Use this to access the original message when working within nested contexts, or to store the entire message as a field. ### [](#examples-4)Examples ```bloblang root.doc = content().string() # In: {"foo":"bar"} # Out: {"doc":"{\"foo\":\"bar\"}"} ``` Preserve original message while adding metadata: ```bloblang root.original = content().string() root.processed_by = "ai" # In: {"foo":"bar"} # Out: {"original":"{\"foo\":\"bar\"}","processed_by":"ai"} ``` ## [](#count)count > ⚠️ **WARNING** > > This function is deprecated and will be removed in a future version. The `count` function is a counter starting at 1 which increments after each time it is called. Count takes an argument which is an identifier for the counter, allowing you to specify multiple unique counters in your configuration. ### [](#parameters-2)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | An identifier for the counter. | ### [](#examples-5)Examples ```bloblang root = this root.id = count("bloblang_function_example") # In: {"message":"foo"} # Out: {"id":1,"message":"foo"} # In: {"message":"bar"} # Out: {"id":2,"message":"bar"} ``` ## [](#counter)counter > ⚠️ **CAUTION** > > This function is experimental. It might change or be removed in a future release. Generates an incrementing sequence of integers starting from a minimum value (default 1). Each counter instance maintains its own independent state across message processing. When the maximum value is reached, the counter automatically resets to the minimum. Introduced in version 4.22.0. ### [](#parameters-3)Parameters | Name | Type | Description | | --- | --- | --- | | min (optional) | query expression | The starting value of the counter. This is the first value yielded. Evaluated once when the mapping is initialized. Default: 1. | | max (optional) | query expression | The maximum value before the counter resets to min. Evaluated once when the mapping is initialized. Default: 9223372036854775807. | | set (optional) | query expression | An optional query that controls counter behavior: when it resolves to a non-negative integer, the counter is set to that value; when it resolves to null, the counter is read without incrementing; when it resolves to a deletion, the counter resets to min; otherwise the counter increments normally. | ### [](#examples-6)Examples Generate sequential IDs for each message: ```bloblang root.id = counter() # In: {} # Out: {"id":1} # In: {} # Out: {"id":2} ``` Use a custom range for the counter: ```bloblang root.batch_num = counter(min: 100, max: 200) # In: {} # Out: {"batch_num":100} # In: {} # Out: {"batch_num":101} ``` Increment a counter multiple times within a single mapping using a named map: ```bloblang map increment { root = counter() } root.first_id = null.apply("increment") root.second_id = null.apply("increment") # In: {} # Out: {"first_id":1,"second_id":2} # In: {} # Out: {"first_id":3,"second_id":4} ``` Conditionally reset a counter based on input data: ```bloblang root.streak = counter(set: if this.status != "success" { 0 }) # In: {"status":"success"} # Out: {"streak":1} # In: {"status":"success"} # Out: {"streak":2} # In: {"status":"failure"} # Out: {"streak":0} # In: {"status":"success"} # Out: {"streak":1} ``` Peek at the current counter value without incrementing by using null in the set parameter: ```bloblang root.count = counter(set: if this.peek { null }) # In: {"peek":false} # Out: {"count":1} # In: {"peek":false} # Out: {"count":2} # In: {"peek":true} # Out: {"count":2} # In: {"peek":false} # Out: {"count":3} ``` ## [](#deleted)deleted Returns a deletion marker that removes the target field or message. When applied to root, the entire message is dropped while still being acknowledged as successfully processed. Use this to filter data or conditionally remove fields. ### [](#examples-7)Examples ```bloblang root = this root.bar = deleted() # In: {"bar":"bar_value","baz":"baz_value","foo":"foo value"} # Out: {"baz":"baz_value","foo":"foo value"} ``` Filter array elements by returning deleted for unwanted items: ```bloblang root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} ``` ## [](#env)env Reads an environment variable and returns its value as a string. Returns `null` if the variable is not set. By default, values are cached for performance. ### [](#parameters-4)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | The name of the environment variable to read. | | no_cache (optional) | bool | Disable caching to read the latest value on each invocation. Default: false. | ### [](#examples-8)Examples ```bloblang root.api_key = env("API_KEY") ``` ```bloblang root.database_url = env("DB_URL").or("localhost:5432") ``` Use `no_cache` to read updated environment variables during runtime, useful for dynamic configuration changes: ```bloblang root.config = env(name: "DYNAMIC_CONFIG", no_cache: true) ``` ## [](#error)error Returns the error message string if the message has failed processing, otherwise `null`. Use this in error handling pipelines to log or route failed messages based on their error details. ### [](#examples-9)Examples ```bloblang root.doc.error = error() ``` Route messages to different outputs based on error presence: ```bloblang root = this root.error_msg = error() root.has_error = error() != null ``` ## [](#error_source_label)error_source_label Returns the user-defined label of the component that caused the error, empty string if no label is set, or `null` if the message has no error. Use this for more human-readable error tracking when components have custom labels. ### [](#examples-10)Examples ```bloblang root.doc.error_source_label = error_source_label() ``` Route errors based on component labels: ```bloblang root.error_category = error_source_label().or("unknown") ``` ## [](#error_source_name)error_source_name Returns the component name that caused the error, or `null` if the message has no error or the error has no associated component. Use this to identify which processor or component in your pipeline caused a failure. ### [](#examples-11)Examples ```bloblang root.doc.error_source_name = error_source_name() ``` Create detailed error logs with component information: ```bloblang root.error_details = if errored() { { "message": error(), "component": error_source_name(), "timestamp": now() } } ``` ## [](#error_source_path)error_source_path Returns the dot-separated path to the component that caused the error, or `null` if the message has no error. Use this to identify the exact location of a failed component in nested pipeline configurations. ### [](#examples-12)Examples ```bloblang root.doc.error_source_path = error_source_path() ``` Build comprehensive error context for debugging: ```bloblang root.error_info = { "path": error_source_path(), "component": error_source_name(), "message": error() } ``` ## [](#errored)errored Returns true if the message has failed processing, false otherwise. Use this for conditional logic in error handling workflows or to route failed messages to dead letter queues. ### [](#examples-13)Examples ```bloblang root.doc.status = if errored() { 400 } else { 200 } ``` Send only failed messages to a separate stream: ```bloblang root = if errored() { this } else { deleted() } ``` ## [](#fake)fake Generates realistic fake data for testing and development purposes. Supports a wide variety of data types including personal information, network addresses, dates/times, financial data, and UUIDs. Useful for creating mock data, populating test databases, or anonymizing sensitive information. Supported functions: `latitude`, `longitude`, `unix_time`, `date`, `time_string`, `month_name`, `year_string`, `day_of_week`, `day_of_month`, `timestamp`, `century`, `timezone`, `time_period`, `email`, `mac_address`, `domain_name`, `url`, `username`, `ipv4`, `ipv6`, `password`, `jwt`, `word`, `sentence`, `paragraph`, `cc_type`, `cc_number`, `currency`, `amount_with_currency`, `title_male`, `title_female`, `first_name`, `first_name_male`, `first_name_female`, `last_name`, `name`, `gender`, `chinese_first_name`, `chinese_last_name`, `chinese_name`, `phone_number`, `toll_free_phone_number`, `e164_phone_number`, `uuid_hyphenated`, `uuid_digit`. Introduced in version 4.4.0. ### [](#parameters-5)Parameters | Name | Type | Description | | --- | --- | --- | | function (optional) | string | The name of the faker function to use. See description for full list of supported functions. Default: "". | ### [](#examples-14)Examples Generate fake user profile data for testing: ```bloblang root.user = { "id": fake("uuid_hyphenated"), "name": fake("name"), "email": fake("email"), "created_at": fake("timestamp") } ``` Create realistic test data for network monitoring: ```bloblang root.event = { "source_ip": fake("ipv4"), "mac_address": fake("mac_address"), "url": fake("url") } ``` ## [](#file)file Reads a file and returns its contents as bytes. Paths are resolved from the process working directory. For paths relative to the mapping file, use `file_rel`. By default, files are cached after first read. ### [](#parameters-6)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | The absolute or relative path to the file. | | no_cache (optional) | bool | Disable caching to read the latest file contents on each invocation. Default: false. | ### [](#examples-15)Examples ```bloblang root.config = file("/etc/config.json").parse_json() ``` ```bloblang root.template = file("./templates/email.html").string() ``` Use `no_cache` to read updated file contents during runtime, useful for hot-reloading configuration: ```bloblang root.rules = file(path: "/etc/rules.yaml", no_cache: true).parse_yaml() ``` ## [](#file_rel)file_rel Reads a file and returns its contents as bytes. Paths are resolved relative to the mapping file’s directory, making it portable across different environments. By default, files are cached after first read. Introduced in version 4.21.0. ### [](#parameters-7)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | The path to the file, relative to the mapping file’s directory. | | no_cache (optional) | bool | Disable caching to read the latest file contents on each invocation. Default: false. | ### [](#examples-16)Examples ```bloblang root.schema = file_rel("./schemas/user.json").parse_json() ``` ```bloblang root.lookup = file_rel("../data/lookup.csv").parse_csv() ``` Use `no_cache` to read updated file contents during runtime, useful for reloading data files without restarting: ```bloblang root.translations = file_rel(path: "./i18n/en.yaml", no_cache: true).parse_yaml() ``` ## [](#hostname)hostname Returns the hostname of the machine running Benthos. Useful for identifying which instance processed a message in distributed deployments. ### [](#examples-17)Examples ```bloblang root.processed_by = hostname() ``` ## [](#json)json Returns a field from the original JSON message by dot path, always accessing the root document regardless of mapping context. Use this to reference the source message when working in nested contexts or to extract specific fields. ### [](#parameters-8)Parameters | Name | Type | Description | | --- | --- | --- | | path (optional) | string | An optional [dot path][field_paths] identifying a field to obtain. Default: "". | ### [](#examples-18)Examples ```bloblang root.mapped = json("foo.bar") # In: {"foo":{"bar":"hello world"}} # Out: {"mapped":"hello world"} ``` Access the original message from within nested mapping contexts: ```bloblang root.doc = json() # In: {"foo":{"bar":"hello world"}} # Out: {"doc":{"foo":{"bar":"hello world"}}} ``` ## [](#ksuid)ksuid Generates a K-Sortable Unique Identifier with built-in timestamp ordering. Use this for distributed unique IDs that sort chronologically and remain collision-resistant without coordination between generators. ### [](#examples-19)Examples ```bloblang root.id = ksuid() ``` Create sortable event IDs for logging: ```bloblang root.event = { "id": ksuid(), "type": this.event_type, "data": this.payload } ``` ## [](#meta)meta > ⚠️ **WARNING** > > This function is deprecated and will be removed in a future version. Returns the value of a metadata key from the input message as a string, or `null` if the key does not exist. Since values are extracted from the read-only input message they do NOT reflect changes made from within the map. In order to query metadata mutations made within a mapping use the [`root_meta` function](#root_meta). This function supports extracting metadata from other messages of a batch with the `from` method. ### [](#parameters-9)Parameters | Name | Type | Description | | --- | --- | --- | | key (optional) | string | An optional key of a metadata value to obtain. Default: "". | ### [](#examples-20)Examples ```bloblang root.topic = meta("kafka_topic") ``` The key parameter is optional and if omitted the entire metadata contents are returned as an object: ```bloblang root.all_metadata = meta() ``` ## [](#metadata)metadata Returns metadata from the input message by key, or `null` if the key doesn’t exist. This reads the original metadata; to access modified metadata during mapping, use the `@` operator instead. Use this to extract message properties like topics, headers, or timestamps. ### [](#parameters-10)Parameters | Name | Type | Description | | --- | --- | --- | | key (optional) | string | An optional key of a metadata value to obtain. Default: "". | ### [](#examples-21)Examples ```bloblang root.topic = metadata("kafka_topic") ``` Retrieve all metadata as an object by omitting the key parameter: ```bloblang root.all_metadata = metadata() ``` Copy specific metadata fields to the message body: ```bloblang root.source = { "topic": metadata("kafka_topic"), "partition": metadata("kafka_partition"), "timestamp": metadata("kafka_timestamp_unix") } ``` ## [](#nanoid)nanoid Generates a URL-safe unique identifier using Nano ID. Use this for compact, URL-friendly IDs with good collision resistance. Customize the length (default 21) or provide a custom alphabet for specific character requirements. ### [](#parameters-11)Parameters | Name | Type | Description | | --- | --- | --- | | length (optional) | integer | An optional length. | | alphabet (optional) | string | An optional custom alphabet to use for generating IDs. When specified the field length must also be present. | ### [](#examples-22)Examples ```bloblang root.id = nanoid() ``` Generate a longer ID for additional uniqueness: ```bloblang root.id = nanoid(54) ``` Use a custom alphabet for domain-specific IDs: ```bloblang root.id = nanoid(54, "abcde") ``` ## [](#now)now Returns the current timestamp as an RFC 3339 formatted string with nanosecond precision. Use this to add processing timestamps to messages or measure time between events. Chain with `ts_format` to customize the format or timezone. ### [](#examples-23)Examples ```bloblang root.received_at = now() ``` Format the timestamp in a custom format and timezone: ```bloblang root.received_at = now().ts_format("Mon Jan 2 15:04:05 -0700 MST 2006", "UTC") ``` ## [](#pi)pi Returns the value of the mathematical constant Pi. Introduced in version 4.30.0. ### [](#examples-24)Examples ```bloblang root.radians = this.degrees * (pi() / 180) # In: {"degrees":45} # Out: {"radians":0.7853981633974483} ``` ```bloblang root.degrees = this.radians * (180 / pi()) # In: {"radians":0.78540} # Out: {"degrees":45.00010522957486} ``` ## [](#random_int)random_int Generates a pseudo-random non-negative 64-bit integer. Use this for creating random IDs, sampling data, or generating test values. Provide a seed for reproducible randomness, or use a dynamic seed like `timestamp_unix_nano()` for unique values per mapping instance. Optional `min` and `max` parameters constrain the output range (both inclusive). For dynamic ranges based on message data, use the modulo operator instead: `random_int() % dynamic_max + dynamic_min`. ### [](#parameters-12)Parameters | Name | Type | Description | | --- | --- | --- | | seed (optional) | query expression | A seed to use, if a query is provided it will only be resolved once during the lifetime of the mapping. | | min (optional) | integer | The minimum value the random generated number will have. The default value is 0. Default: 0. | | max (optional) | integer | The maximum value the random generated number will have. The default value is 9223372036854775806 (math.MaxInt64 - 1). Default: 9223372036854775806. | ### [](#examples-25)Examples ```bloblang root.first = random_int() root.second = random_int(1) root.third = random_int(max:20) root.fourth = random_int(min:10, max:20) root.fifth = random_int(timestamp_unix_nano(), 5, 20) root.sixth = random_int(seed:timestamp_unix_nano(), max:20) ``` Use a dynamic seed for unique random values per mapping instance: ```bloblang root.random_id = random_int(timestamp_unix_nano()) root.sample_percent = random_int(seed: timestamp_unix_nano(), min: 0, max: 100) ``` ## [](#range)range Creates an array of integers from start (inclusive) to stop (exclusive) with an optional step. Use this to generate sequences for iteration, indexing, or creating numbered lists. ### [](#parameters-13)Parameters | Name | Type | Description | | --- | --- | --- | | start | integer | The start value. | | stop | integer | The stop value. | | step (optional) | integer | The step value. Default: 1. | ### [](#examples-26)Examples ```bloblang root.a = range(0, 10) root.b = range(start: 0, stop: this.max, step: 2) # Using named params root.c = range(0, -this.max, -2) # In: {"max":10} # Out: {"a":[0,1,2,3,4,5,6,7,8,9],"b":[0,2,4,6,8],"c":[0,-2,-4,-6,-8]} ``` Generate a sequence for batch processing: ```bloblang root.pages = range(0, this.total_items, 100).map_each(offset -> { "offset": offset, "limit": 100 }) # In: {"total_items":250} # Out: {"pages":[{"limit":100,"offset":0},{"limit":100,"offset":100}]} ``` ## [](#root_meta)root_meta > ⚠️ **WARNING** > > This function is deprecated and will be removed in a future version. Returns the value of a metadata key from the new message being created as a string, or `null` if the key does not exist. Changes made to metadata during a mapping will be reflected by this function. ### [](#parameters-14)Parameters | Name | Type | Description | | --- | --- | --- | | key (optional) | string | An optional key of a metadata value to obtain. Default: "". | ### [](#examples-27)Examples ```bloblang root.topic = root_meta("kafka_topic") ``` The key parameter is optional and if omitted the entire metadata contents are returned as an object: ```bloblang root.all_metadata = root_meta() ``` ## [](#snowflake_id)snowflake_id Generates a unique, time-ordered Snowflake ID. Snowflake IDs are 64-bit integers that encode timestamp, node ID, and sequence information, making them ideal for distributed systems where sortable unique identifiers are needed. Returns a string representation of the ID. Introduced in version 4.6.0. ### [](#parameters-15)Parameters | Name | Type | Description | | --- | --- | --- | | node_id (optional) | integer | Optional node identifier (0-1023) to distinguish IDs generated by different machines in a distributed system. Defaults to 1. Default: 1. | ### [](#examples-28)Examples Generate a unique Snowflake ID for each message: ```bloblang root.id = snowflake_id() root.payload = this ``` Generate Snowflake IDs with different node IDs for multi-datacenter deployments: ```bloblang root.id = snowflake_id(42) root.data = this ``` ## [](#throw)throw Immediately fails the mapping with a custom error message. Use this to halt processing when data validation fails or required fields are missing, causing the message to be routed to error handlers. ### [](#parameters-16)Parameters | Name | Type | Description | | --- | --- | --- | | why | string | A string explanation for why an error was thrown, this will be added to the resulting error message. | ### [](#examples-29)Examples ```bloblang root.doc.type = match { this.exists("header.id") => "foo" this.exists("body.data") => "bar" _ => throw("unknown type") } root.doc.contents = (this.body.content | this.thing.body) # In: {"header":{"id":"first"},"thing":{"body":"hello world"}} # Out: {"doc":{"contents":"hello world","type":"foo"}} # In: {"nothing":"matches"} # Out: Error("failed assignment (line 1): unknown type") ``` Validate required fields before processing: ```bloblang root = if this.exists("user_id") { this } else { throw("missing required field: user_id") } # In: {"user_id":123,"name":"alice"} # Out: {"name":"alice","user_id":123} # In: {"name":"bob"} # Out: Error("failed assignment (line 1): missing required field: user_id") ``` ## [](#timestamp_unix)timestamp_unix Returns the current Unix timestamp in seconds since epoch. Use this for numeric timestamps compatible with most systems, or as a seed for random number generation. ### [](#examples-30)Examples ```bloblang root.received_at = timestamp_unix() ``` Create a sortable ID combining timestamp with a counter: ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` ## [](#timestamp_unix_micro)timestamp_unix_micro Returns the current Unix timestamp in microseconds since epoch. Use this for high-precision timing measurements or when microsecond resolution is required. ### [](#examples-31)Examples ```bloblang root.received_at = timestamp_unix_micro() ``` Measure elapsed time between events: ```bloblang root.processing_duration_us = timestamp_unix_micro() - this.start_time_us ``` ## [](#timestamp_unix_milli)timestamp_unix_milli Returns the current Unix timestamp in milliseconds since epoch. Use this for millisecond-precision timestamps common in web APIs and JavaScript systems. ### [](#examples-32)Examples ```bloblang root.received_at = timestamp_unix_milli() ``` Add processing time metadata: ```bloblang meta processing_time_ms = timestamp_unix_milli() ``` ## [](#timestamp_unix_nano)timestamp_unix_nano Returns the current Unix timestamp in nanoseconds since epoch. Use this for the highest precision timing or as a unique seed value that changes on every invocation. ### [](#examples-33)Examples ```bloblang root.received_at = timestamp_unix_nano() ``` Generate unique random values on each mapping: ```bloblang root.random_value = random_int(timestamp_unix_nano()) ``` ## [](#tracing_id)tracing_id > ⚠️ **CAUTION** > > This function is experimental. It might change or be removed in a future release. Returns the OpenTelemetry trace ID for the message, or an empty string if no tracing span exists. Use this to correlate logs and events with distributed traces. ### [](#examples-34)Examples ```bloblang meta trace_id = tracing_id() ``` Add trace ID to structured logs: ```bloblang root.log_entry = this root.log_entry.trace_id = tracing_id() ``` ## [](#tracing_span)tracing_span > ⚠️ **CAUTION** > > This function is experimental. It might change or be removed in a future release. Returns the OpenTelemetry tracing span attached to the message as a text map object, or `null` if no span exists. Use this to propagate trace context to downstream systems via headers or metadata. ### [](#examples-35)Examples ```bloblang root.headers.traceparent = tracing_span().traceparent # In: {"some_stuff":"just can't be explained by science"} # Out: {"headers":{"traceparent":"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"}} ``` Forward all tracing fields to output metadata: ```bloblang meta = tracing_span() ``` ## [](#ulid)ulid Generates a Universally Unique Lexicographically Sortable Identifier (ULID). ULIDs are 128-bit identifiers that are sortable by creation time, URL-safe, and case-insensitive. They consist of a 48-bit timestamp (millisecond precision) and 80 bits of randomness, making them ideal for distributed systems that need time-ordered unique IDs without coordination. Introduced in version 4.16.0. ### [](#parameters-17)Parameters | Name | Type | Description | | --- | --- | --- | | encoding (optional) | string | Encoding format for the ULID. "crockford" produces 26-character Base32 strings (recommended). "hex" produces 32-character hexadecimal strings. Default: "crockford". | | random_source (optional) | string | Randomness source: "secure_random" uses cryptographically secure random (recommended for production), "fast_random" uses faster but non-secure random (only for non-sensitive testing). Default: "secure_random". | ### [](#examples-36)Examples Generate time-sortable IDs for distributed message ordering: ```bloblang root.message_id = ulid() root.timestamp = now() root.data = this ``` Generate hex-encoded ULIDs for systems that prefer hexadecimal format: ```bloblang root.id = ulid("hex") ``` ## [](#uuid_v4)uuid_v4 Generates a random RFC-4122 version 4 UUID. Use this for creating unique identifiers that don’t reveal timing information or require ordering. Each invocation produces a new globally unique ID. ### [](#examples-37)Examples ```bloblang root.id = uuid_v4() ``` Add unique request IDs for tracing: ```bloblang root = this root.request_id = uuid_v4() ``` ## [](#uuid_v7)uuid_v7 Generates a time-ordered UUID version 7 with millisecond timestamp precision. Use this for sortable unique identifiers that maintain chronological ordering, ideal for database keys or event IDs. Optionally specify a custom timestamp. ### [](#parameters-18)Parameters | Name | Type | Description | | --- | --- | --- | | time (optional) | timestamp | An optional timestamp to use for the time ordered portion of the UUID. | ### [](#examples-38)Examples ```bloblang root.id = uuid_v7() ``` Generate a UUID with a specific timestamp for backdating events: ```bloblang root.id = uuid_v7(now().ts_sub_iso8601("PT1M")) ``` ## [](#with_schema_registry_header)with_schema_registry_header Prepends a Confluent Schema Registry wire format header to message bytes. The header is 5 bytes: a magic byte (0x00) followed by a 4-byte big-endian schema ID. This format is required when producing messages to Kafka topics that use Confluent Schema Registry for schema validation and evolution. ### [](#parameters-19)Parameters | Name | Type | Description | | --- | --- | --- | | schema_id | unknown | The schema ID from your Schema Registry (0 to 4294967295). This ID references the schema version used to encode the message. | | message | unknown | The serialized message bytes (for example, Avro, Protobuf, or JSON Schema encoded data) to prepend the header to. | ### [](#examples-39)Examples Add Schema Registry header to Avro-encoded message: ```bloblang root = with_schema_registry_header(123, content()) ``` Use schema ID from metadata to add header dynamically: ```bloblang root = with_schema_registry_header(meta("schema_id").number(), content()) ``` --- # Page 380: Bloblang Methods **URL**: https://docs.redpanda.com/connect/guides/bloblang/methods.md --- # Bloblang Methods --- title: Bloblang Methods latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/methods page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/methods.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/methods.adoc description: A list of Bloblang methods page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- Methods provide most of the power in Bloblang as they allow you to augment values and can be added to any expression (including other methods): ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,18,7,25]},"user":{"fav_pokemon":"pikachu"}} ``` Methods support both named and nameless style arguments: ```bloblang root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} ``` ## [](#general)General ### [](#apply)apply Apply a declared mapping to a target value. #### [](#parameters)Parameters | Name | Type | Description | | --- | --- | --- | | mapping | string | The mapping to apply. | #### [](#examples)Examples ```bloblang map thing { root.inner = this.first } root.foo = this.doc.apply("thing") # In: {"doc":{"first":"hello world"}} # Out: {"foo":{"inner":"hello world"}} ``` ```bloblang map create_foo { root.name = "a foo" root.purpose = "to be a foo" } root = this root.foo = null.apply("create_foo") # In: {"id":"1234"} # Out: {"foo":{"name":"a foo","purpose":"to be a foo"},"id":"1234"} ``` ### [](#catch)catch If the result of a target query fails (due to incorrect types, failed parsing, etc) the argument is returned instead. #### [](#parameters-2)Parameters | Name | Type | Description | | --- | --- | --- | | fallback | query expression | A value to yield, or query to execute, if the target query fails. | #### [](#examples-2)Examples ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) ``` The fallback argument can be a mapping, allowing you to capture the error string and yield structured data back: ```bloblang root.url = this.url.parse_url().catch(err -> {"error":err,"input":this.url}) # In: {"url":"invalid %&# url"} # Out: {"url":{"error":"field `this.url`: parse \"invalid %&\": invalid URL escape \"%&\"","input":"invalid %&# url"}} ``` When the input document is not structured attempting to reference structured fields with `this` will result in an error. Therefore, a convenient way to delete non-structured data is with a catch: ```bloblang root = this.catch(deleted()) # In: {"doc":{"foo":"bar"}} # Out: {"doc":{"foo":"bar"}} # In: not structured data # Out: ``` ### [](#from)from Modifies a target query such that certain functions are executed from the perspective of another message in the batch. This allows you to mutate events based on the contents of other messages. Functions that support this behavior are `content`, `json` and `meta`. #### [](#parameters-3)Parameters | Name | Type | Description | | --- | --- | --- | | index | integer | The message index to use as a perspective. | #### [](#examples-3)Examples For example, the following map extracts the contents of the JSON field `foo` specifically from message index `1` of a batch, effectively overriding the field `foo` for all messages of a batch to that of message 1: ```bloblang root = this root.foo = json("foo").from(1) ``` ### [](#from_all)from_all Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Functions that support this behavior are `content`, `json` and `meta`. #### [](#examples-4)Examples ```bloblang root = this root.foo_summed = json("foo").from_all().sum() ``` ### [](#map)map Executes a query on the target value, allowing you to transform or extract data from the current context. #### [](#parameters-4)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute on the target. | ### [](#not)not Returns the logical NOT (negation) of a boolean value. Converts true to false and false to true. ### [](#or)or If the result of the target query fails or resolves to `null`, returns the argument instead. This is an explicit method alternative to the coalesce pipe operator `|`. #### [](#parameters-5)Parameters | Name | Type | Description | | --- | --- | --- | | fallback | query expression | A value to yield, or query to execute, if the target query fails or resolves to null. | #### [](#examples-5)Examples ```bloblang root.doc.id = this.thing.id.or(uuid_v4()) ``` ## [](#encoding-and-encryption)Encoding and encryption ### [](#compress)compress Compresses a string or byte array using the specified compression algorithm. Returns compressed data as bytes. Useful for reducing payload size before transmission or storage. Introduced in version 4.12.0. #### [](#parameters-6)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The compression algorithm: flate, gzip, pgzip (parallel gzip), lz4, snappy, zlib, or zstd. | | level (optional) | integer | Compression level (default: -1 for default compression). Higher values increase compression ratio but use more CPU. Range and effect varies by algorithm. Default: -1. | #### [](#examples-6)Examples Compress and encode for safe transmission: ```bloblang root.compressed = content().bytes().compress("gzip").encode("base64") # In: {"message":"hello world I love space"} # Out: {"compressed":"H4sIAAAJbogA/wAmANn/eyJtZXNzYWdlIjoiaGVsbG8gd29ybGQgSSBsb3ZlIHNwYWNlIn0DAHEvdwomAAAA"} ``` Compare compression ratios across algorithms: ```bloblang root.original_size = content().length() root.gzip_size = content().compress("gzip").length() root.lz4_size = content().compress("lz4").length() # In: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. # Out: {"gzip_size":114,"lz4_size":85,"original_size":89} ``` ### [](#decode)decode Decodes an encoded string target according to a chosen scheme and returns the result as a byte array. When mapping the result to a JSON field the value should be cast to a string using the method `string`, or encoded using the method `encode`, otherwise it will be base64 encoded by default. Available schemes are: `base64`, `base64url` [(RFC 4648 with padding characters)](https://rfc-editor.org/rfc/rfc4648.html), `base64rawurl` [(RFC 4648 without padding characters)](https://rfc-editor.org/rfc/rfc4648.html), `hex`, `ascii85`. #### [](#parameters-7)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The decoding scheme to use. | #### [](#examples-7)Examples ```bloblang root.decoded = this.value.decode("hex").string() # In: {"value":"68656c6c6f20776f726c64"} # Out: {"decoded":"hello world"} ``` ```bloblang root = this.encoded.decode("ascii85") # In: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} # Out: this is totally unstructured data ``` ### [](#decompress)decompress Decompresses a byte array using the specified decompression algorithm. Returns decompressed data as bytes. Use with data that was previously compressed using the corresponding algorithm. Introduced in version 4.12.0. #### [](#parameters-8)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The decompression algorithm: gzip, pgzip (parallel gzip), zlib, bzip2, flate, snappy, lz4, or zstd. | #### [](#examples-8)Examples Decompress base64-encoded compressed data: ```bloblang root = this.compressed.decode("base64").decompress("gzip") # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: hello world I love space ``` Convert decompressed bytes to string for JSON output: ```bloblang root.message = this.compressed.decode("base64").decompress("gzip").string() # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: {"message":"hello world I love space"} ``` ### [](#decrypt_aes)decrypt_aes Decrypts an encrypted string or byte array target according to a chosen AES encryption method and returns the result as a byte array. The algorithms require a key and an initialization vector / nonce. Available schemes are: `ctr`, `gcm`, `ofb`, `cbc`. #### [](#parameters-9)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The scheme to use for decryption, one of ctr, gcm, ofb, cbc. | | key | string | A key to decrypt with. | | iv | string | An initialization vector / nonce. | #### [](#examples-9)Examples ```bloblang let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.decrypted = this.value.decode("hex").decrypt_aes("ctr", $key, $vector).string() # In: {"value":"84e9b31ff7400bdf80be7254"} # Out: {"decrypted":"hello world!"} ``` ### [](#encode)encode Encodes a string or byte array target according to a chosen scheme and returns a string result. Available schemes are: `base64`, `base64url` [(RFC 4648 with padding characters)](https://rfc-editor.org/rfc/rfc4648.html), `base64rawurl` [(RFC 4648 without padding characters)](https://rfc-editor.org/rfc/rfc4648.html), `hex`, `ascii85`. #### [](#parameters-10)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The encoding scheme to use. | #### [](#examples-10)Examples ```bloblang root.encoded = this.value.encode("hex") # In: {"value":"hello world"} # Out: {"encoded":"68656c6c6f20776f726c64"} ``` ```bloblang root.encoded = content().encode("ascii85") # In: this is totally unstructured data # Out: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} ``` ### [](#encrypt_aes)encrypt_aes Encrypts a string or byte array target according to a chosen AES encryption method and returns a string result. The algorithms require a key and an initialization vector / nonce. Available schemes are: `ctr`, `gcm`, `ofb`, `cbc`. #### [](#parameters-11)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The scheme to use for encryption, one of ctr, gcm, ofb, cbc. | | key | string | A key to encrypt with. | | iv | string | An initialization vector / nonce. | #### [](#examples-11)Examples ```bloblang let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.encrypted = this.value.encrypt_aes("ctr", $key, $vector).encode("hex") # In: {"value":"hello world!"} # Out: {"encrypted":"84e9b31ff7400bdf80be7254"} ``` ### [](#hash)hash Hashes a string or byte array according to a chosen algorithm and returns the result as a byte array. When mapping the result to a JSON field the value should be cast to a string using the method [`string`](#string), or encoded using the method [`encode`](#encode), otherwise it will be base64 encoded by default. Available algorithms are: `hmac_sha1`, `hmac_sha256`, `hmac_sha512`, `md5`, `sha1`, `sha256`, `sha512`, `sha3_256`, `sha3_512`, `xxhash64`, `crc32`, `fnv32`. The following algorithms require a key, which is specified as a second argument: `hmac_sha1`, `hmac_sha256`, `hmac_sha512`. #### [](#parameters-12)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The hashing algorithm to use. | | key (optional) | string | An optional key to use. | | polynomial (optional) | string | An optional polynomial key to use when selecting the crc32 algorithm, otherwise ignored. Options are IEEE (default), Castagnoli and Koopman. Default: "IEEE". | #### [](#examples-12)Examples ```bloblang root.h1 = this.value.hash("sha1").encode("hex") root.h2 = this.value.hash("hmac_sha1","static-key").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"2aae6c35c94fcfb415dbe95f408b9ce91ee846ed","h2":"d87e5f068fa08fe90bb95bc7c8344cb809179d76"} ``` The `crc32` algorithm supports options for the polynomial: ```bloblang root.h1 = this.value.hash(algorithm: "crc32", polynomial: "Castagnoli").encode("hex") root.h2 = this.value.hash(algorithm: "crc32", polynomial: "Koopman").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"c99465aa","h2":"df373d3c"} ``` ### [](#uuid_v5)uuid_v5 Returns UUID version 5 for the given string. #### [](#parameters-13)Parameters | Name | Type | Description | | --- | --- | --- | | ns (optional) | string | An optional namespace name or UUID. It supports the dns, url, oid and x500 predefined namespaces and any valid RFC-9562 UUID. If empty, the nil UUID will be used. | #### [](#examples-13)Examples ```bloblang root.id = "example".uuid_v5() ``` ```bloblang root.id = "example".uuid_v5("x500") ``` ```bloblang root.id = "example".uuid_v5("77f836b7-9f61-46c0-851e-9b6ca3535e69") ``` ## [](#geoip)GeoIP ### [](#geoip_anonymous_ip)geoip_anonymous_ip Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the anonymous IP associated with it. #### [](#parameters-14)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_asn)geoip_asn Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the ASN associated with it. #### [](#parameters-15)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_city)geoip_city Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the city associated with it. #### [](#parameters-16)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_connection_type)geoip_connection_type Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the connection type associated with it. #### [](#parameters-17)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_country)geoip_country Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the country associated with it. #### [](#parameters-18)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_domain)geoip_domain Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the domain associated with it. #### [](#parameters-19)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_enterprise)geoip_enterprise Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the enterprise associated with it. #### [](#parameters-20)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_isp)geoip_isp Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the ISP associated with it. #### [](#parameters-21)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ## [](#json-web-tokens)JSON web tokens ### [](#parse_jwt_es256)parse_jwt_es256 Parses a claims object from a JWT string encoded with ES256. This method does not validate JWT claims. Introduced in version 4.20.0. #### [](#parameters-22)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES256 secret that was used for signing the token. | #### [](#examples-14)Examples ```bloblang root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGtLqIBePHmIhQcf0JLgc+F/4W/oI dp0Gta53G35VerNDgUUXmp78J2kfh4qLdh0XtmOMI587tCaqjvDAXfs//w== -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.GIRajP9JJbpTlqSCdNEz4qpQkRvzX4Q51YnTwVyxLDM9tKjR_a8ggHWn9CWj7KG0x8J56OWtmUxn112SRTZVhQ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_es384)parse_jwt_es384 Parses a claims object from a JWT string encoded with ES384. This method does not validate JWT claims. Introduced in version 4.20.0. #### [](#parameters-23)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES384 secret that was used for signing the token. | #### [](#examples-15)Examples ```bloblang root.claims = this.signed.parse_jwt_es384("""-----BEGIN PUBLIC KEY----- MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERoz74/B6SwmLhs8X7CWhnrWyRrB13AuU 8OYeqy0qHRu9JWNw8NIavqpTmu6XPT4xcFanYjq8FbeuM11eq06C52mNmS4LLwzA 2imlFEgn85bvJoC3bnkuq4mQjwt9VxdH -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.H2HBSlrvQBaov2tdreGonbBexxtQB-xzaPL4-tNQZ6TVh7VH8VBcSwcWHYa1lBAHmdsKOFcB2Wk0SB7QWeGT3ptSgr-_EhDMaZ8bA5spgdpq5DsKfaKHrd7DbbQlmxNq"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_es512)parse_jwt_es512 Parses a claims object from a JWT string encoded with ES512. This method does not validate JWT claims. Introduced in version 4.20.0. #### [](#parameters-24)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES512 secret that was used for signing the token. | #### [](#examples-16)Examples ```bloblang root.claims = this.signed.parse_jwt_es512("""-----BEGIN PUBLIC KEY----- MIGbMBAGByqGSM49AgEGBSuBBAAjA4GGAAQAkHLdts9P56fFkyhpYQ31M/Stwt3w vpaxhlfudxnXgTO1IP4RQRgryRxZ19EUzhvWDcG3GQIckoNMY5PelsnCGnIBT2Xh 9NQkjWF5K6xS4upFsbGSAwQ+GIyyk5IPJ2LHgOyMSCVh5gRZXV3CZLzXujx/umC9 UeYyTt05zRRWuD+p5bY= -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.ACrpLuU7TKpAnncDCpN9m85nkL55MJ45NFOBl6-nEXmNT1eIxWjiP4pwWVbFH9et_BgN14119jbL_KqEJInPYc9nAXC6dDLq0aBU-dalvNl4-O5YWpP43-Y-TBGAsWnbMTrchILJ4-AEiICe73Ck5yWPleKg9c3LtkEFWfGs7BoPRguZ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs256)parse_jwt_hs256 Parses a claims object from a JWT string encoded with HS256. This method does not validate JWT claims. Introduced in version 4.12.0. #### [](#parameters-25)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS256 secret that was used for signing the token. | #### [](#examples-17)Examples ```bloblang root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.YwXOM8v3gHVWcQRRRQc_zDlhmLnM62fwhFYGpiA0J1A"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs384)parse_jwt_hs384 Parses a claims object from a JWT string encoded with HS384. This method does not validate JWT claims. Introduced in version 4.12.0. #### [](#parameters-26)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS384 secret that was used for signing the token. | #### [](#examples-18)Examples ```bloblang root.claims = this.signed.parse_jwt_hs384("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.2Y8rf_ijwN4t8hOGGViON_GrirLkCQVbCOuax6EoZ3nluX0tCGezcJxbctlIfsQ2"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs512)parse_jwt_hs512 Parses a claims object from a JWT string encoded with HS512. This method does not validate JWT claims. Introduced in version 4.12.0. #### [](#parameters-27)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS512 secret that was used for signing the token. | #### [](#examples-19)Examples ```bloblang root.claims = this.signed.parse_jwt_hs512("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.utRb0urG6LGGyranZJVo5Dk0Fns1QNcSUYPN0TObQ-YzsGGB8jrxHwM5NAJccjJZzKectEUqmmKCaETZvuX4Fg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs256)parse_jwt_rs256 Parses a claims object from a JWT string encoded with RS256. This method does not validate JWT claims. Introduced in version 4.20.0. #### [](#parameters-28)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS256 secret that was used for signing the token. | #### [](#examples-20)Examples ```bloblang root.claims = this.signed.parse_jwt_rs256("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs384)parse_jwt_rs384 Parses a claims object from a JWT string encoded with RS384. This method does not validate JWT claims. Introduced in version 4.20.0. #### [](#parameters-29)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS384 secret that was used for signing the token. | #### [](#examples-21)Examples ```bloblang root.claims = this.signed.parse_jwt_rs384("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs512)parse_jwt_rs512 Parses a claims object from a JWT string encoded with RS512. This method does not validate JWT claims. Introduced in version 4.20.0. #### [](#parameters-30)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS512 secret that was used for signing the token. | #### [](#examples-22)Examples ```bloblang root.claims = this.signed.parse_jwt_rs512("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#sign_jwt_es256)sign_jwt_es256 Hash and sign an object representing JSON Web Token (JWT) claims using ES256. Introduced in version 4.20.0. #### [](#parameters-31)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-23)Examples ```bloblang root.signed = this.claims.sign_jwt_es256("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_es256(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_es384)sign_jwt_es384 Hash and sign an object representing JSON Web Token (JWT) claims using ES384. Introduced in version 4.20.0. #### [](#parameters-32)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-24)Examples ```bloblang root.signed = this.claims.sign_jwt_es384("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_es384(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_es512)sign_jwt_es512 Hash and sign an object representing JSON Web Token (JWT) claims using ES512. Introduced in version 4.20.0. #### [](#parameters-33)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-25)Examples ```bloblang root.signed = this.claims.sign_jwt_es512("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_es512(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs256)sign_jwt_hs256 Hash and sign an object representing JSON Web Token (JWT) claims using HS256. Introduced in version 4.12.0. #### [](#parameters-34)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-26)Examples ```bloblang root.signed = this.claims.sign_jwt_hs256("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_hs256(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs384)sign_jwt_hs384 Hash and sign an object representing JSON Web Token (JWT) claims using HS384. Introduced in version 4.12.0. #### [](#parameters-35)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-27)Examples ```bloblang root.signed = this.claims.sign_jwt_hs384("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_hs384(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs512)sign_jwt_hs512 Hash and sign an object representing JSON Web Token (JWT) claims using HS512. Introduced in version 4.12.0. #### [](#parameters-36)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-28)Examples ```bloblang root.signed = this.claims.sign_jwt_hs512("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_hs512(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs256)sign_jwt_rs256 Hash and sign an object representing JSON Web Token (JWT) claims using RS256. Introduced in version 4.18.0. #### [](#parameters-37)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-29)Examples ```bloblang root.signed = this.claims.sign_jwt_rs256("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_rs256(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs384)sign_jwt_rs384 Hash and sign an object representing JSON Web Token (JWT) claims using RS384. Introduced in version 4.18.0. #### [](#parameters-38)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-30)Examples ```bloblang root.signed = this.claims.sign_jwt_rs384("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_rs384(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs512)sign_jwt_rs512 Hash and sign an object representing JSON Web Token (JWT) claims using RS512. Introduced in version 4.18.0. #### [](#parameters-39)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-31)Examples ```bloblang root.signed = this.claims.sign_jwt_rs512("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ```bloblang root.signed = this.claims.sign_jwt_rs512(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ## [](#number-manipulation)Number manipulation ### [](#abs)abs Returns the absolute value of an int64 or float64 number. As a special case, when an integer is provided that is the minimum value it is converted to the maximum value. #### [](#examples-32)Examples ```bloblang root.outs = this.ins.map_each(ele -> ele.abs()) # In: {"ins":[9,-18,1.23,-4.56]} # Out: {"outs":[9,18,1.23,4.56]} ``` ### [](#bitwise_and)bitwise_and Performs a bitwise AND operation between the integer and the specified value. #### [](#parameters-40)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to AND with | #### [](#examples-33)Examples ```bloblang root.new_value = this.value.bitwise_and(6) # In: {"value":12} # Out: {"new_value":4} ``` ```bloblang root.masked = this.flags.bitwise_and(15) # In: {"flags":127} # Out: {"masked":15} ``` ### [](#bitwise_or)bitwise_or Performs a bitwise OR operation between the integer and the specified value. #### [](#parameters-41)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to OR with | #### [](#examples-34)Examples ```bloblang root.new_value = this.value.bitwise_or(6) # In: {"value":12} # Out: {"new_value":14} ``` ```bloblang root.combined = this.flags.bitwise_or(8) # In: {"flags":4} # Out: {"combined":12} ``` ### [](#bitwise_xor)bitwise_xor Performs a bitwise XOR (exclusive OR) operation between the integer and the specified value. #### [](#parameters-42)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to XOR with | #### [](#examples-35)Examples ```bloblang root.new_value = this.value.bitwise_xor(6) # In: {"value":12} # Out: {"new_value":10} ``` ```bloblang root.toggled = this.flags.bitwise_xor(5) # In: {"flags":3} # Out: {"toggled":6} ``` ### [](#ceil)ceil Rounds a number up to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-36)Examples ```bloblang root.new_value = this.value.ceil() # In: {"value":5.3} # Out: {"new_value":6} # In: {"value":-5.9} # Out: {"new_value":-5} ``` ```bloblang root.result = this.price.ceil() # In: {"price":19.99} # Out: {"result":20} ``` ### [](#cos)cos Calculates the cosine of a given angle specified in radians. Introduced in version 4.30.0. #### [](#examples-37)Examples ```bloblang root.new_value = (this.value * (pi() / 180)).cos() # In: {"value":45} # Out: {"new_value":0.7071067811865476} # In: {"value":0} # Out: {"new_value":1} # In: {"value":180} # Out: {"new_value":-1} ``` ### [](#float32)float32 Converts a numerical type into a 32-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit floating point number. Please refer to the [`strconv.ParseFloat` documentation](https://pkg.go.dev/strconv#ParseFloat) for details regarding the supported formats. Introduced in version 4.19.0. #### [](#examples-38)Examples ```bloblang root.out = this.in.float32() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674283e-11} ``` ### [](#float64)float64 Converts a numerical type into a 64-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit floating point number. Please refer to the [`strconv.ParseFloat` documentation](https://pkg.go.dev/strconv#ParseFloat) for details regarding the supported formats. Introduced in version 4.19.0. #### [](#examples-39)Examples ```bloblang root.out = this.in.float64() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674282313423544e-11} ``` ### [](#floor)floor Rounds a number down to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-40)Examples ```bloblang root.new_value = this.value.floor() # In: {"value":5.7} # Out: {"new_value":5} # In: {"value":-3.2} # Out: {"new_value":-4} ``` ```bloblang root.whole_seconds = this.duration_seconds.floor() # In: {"duration_seconds":12.345} # Out: {"whole_seconds":12} ``` ### [](#int16)int16 Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-41)Examples ```bloblang root.a = this.a.int16() root.b = this.b.round().int16() root.c = this.c.int16() root.d = this.d.int16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int16() # In: "0xDE" # Out: 222 ``` ### [](#int32)int32 Converts a numerical type into a 32-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-42)Examples ```bloblang root.a = this.a.int32() root.b = this.b.round().int32() root.c = this.c.int32() root.d = this.d.int32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int32() # In: "0xDEAD" # Out: 57005 ``` ### [](#int64)int64 Converts a numerical type into a 64-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-43)Examples ```bloblang root.a = this.a.int64() root.b = this.b.round().int64() root.c = this.c.int64() root.d = this.d.int64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int64() # In: "0xDEADBEEF" # Out: 3735928559 ``` ### [](#int8)int8 Converts a numerical type into a 8-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-44)Examples ```bloblang root.a = this.a.int8() root.b = this.b.round().int8() root.c = this.c.int8() root.d = this.d.int8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int8() # In: "0xD" # Out: 13 ``` ### [](#log)log Calculates the natural logarithm (base e) of a number. #### [](#examples-45)Examples ```bloblang root.new_value = this.value.log().round() # In: {"value":1} # Out: {"new_value":0} # In: {"value":2.7183} # Out: {"new_value":1} ``` ```bloblang root.ln_result = this.number.log() # In: {"number":10} # Out: {"ln_result":2.302585092994046} ``` ### [](#log10)log10 Calculates the base-10 logarithm of a number. #### [](#examples-46)Examples ```bloblang root.new_value = this.value.log10() # In: {"value":100} # Out: {"new_value":2} # In: {"value":1000} # Out: {"new_value":3} ``` ```bloblang root.log_value = this.magnitude.log10() # In: {"magnitude":10000} # Out: {"log_value":4} ``` ### [](#max)max Returns the largest number from an array. All elements must be numbers and the array cannot be empty. #### [](#examples-47)Examples ```bloblang root.biggest = this.values.max() # In: {"values":[0,3,2.5,7,5]} # Out: {"biggest":7} ``` ```bloblang root.highest_temp = this.temperatures.max() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"highest_temp":23.4} ``` ### [](#min)min Returns the smallest number from an array. All elements must be numbers and the array cannot be empty. #### [](#examples-48)Examples ```bloblang root.smallest = this.values.min() # In: {"values":[0,3,-2.5,7,5]} # Out: {"smallest":-2.5} ``` ```bloblang root.lowest_temp = this.temperatures.min() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"lowest_temp":19.8} ``` ### [](#pow)pow Returns the number raised to the specified exponent. Introduced in version 4.30.0. #### [](#parameters-43)Parameters | Name | Type | Description | | --- | --- | --- | | exponent | float | The exponent you want to raise to the power of. | #### [](#examples-49)Examples ```bloblang root.new_value = this.value * 10.pow(-2) # In: {"value":2} # Out: {"new_value":0.02} ``` ```bloblang root.new_value = this.value.pow(-2) # In: {"value":2} # Out: {"new_value":0.25} ``` ### [](#round)round Rounds a number to the nearest integer. Values at .5 round away from zero. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-50)Examples ```bloblang root.new_value = this.value.round() # In: {"value":5.3} # Out: {"new_value":5} # In: {"value":5.9} # Out: {"new_value":6} ``` ```bloblang root.rounded = this.score.round() # In: {"score":87.5} # Out: {"rounded":88} ``` ### [](#sin)sin Calculates the sine of a given angle specified in radians. Introduced in version 4.30.0. #### [](#examples-51)Examples ```bloblang root.new_value = (this.value * (pi() / 180)).sin() # In: {"value":45} # Out: {"new_value":0.7071067811865475} # In: {"value":0} # Out: {"new_value":0} # In: {"value":90} # Out: {"new_value":1} ``` ### [](#tan)tan Calculates the tangent of a given angle specified in radians. Introduced in version 4.30.0. #### [](#examples-52)Examples ```bloblang root.new_value = "%f".format((this.value * (pi() / 180)).tan()) # In: {"value":0} # Out: {"new_value":"0.000000"} # In: {"value":45} # Out: {"new_value":"1.000000"} # In: {"value":180} # Out: {"new_value":"-0.000000"} ``` ### [](#uint16)uint16 Converts a numerical type into a 16-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-53)Examples ```bloblang root.a = this.a.uint16() root.b = this.b.round().uint16() root.c = this.c.uint16() root.d = this.d.uint16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint16() # In: "0xDE" # Out: 222 ``` ### [](#uint32)uint32 Converts a numerical type into a 32-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-54)Examples ```bloblang root.a = this.a.uint32() root.b = this.b.round().uint32() root.c = this.c.uint32() root.d = this.d.uint32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint32() # In: "0xDEAD" # Out: 57005 ``` ### [](#uint64)uint64 Converts a numerical type into a 64-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-55)Examples ```bloblang root.a = this.a.uint64() root.b = this.b.round().uint64() root.c = this.c.uint64() root.d = this.d.uint64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint64() # In: "0xDEADBEEF" # Out: 3735928559 ``` ### [](#uint8)uint8 Converts a numerical type into a 8-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-56)Examples ```bloblang root.a = this.a.uint8() root.b = this.b.round().uint8() root.c = this.c.uint8() root.d = this.d.uint8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint8() # In: "0xD" # Out: 13 ``` ## [](#object-array-manipulation)Object & array manipulation ### [](#all)all Tests whether all elements in an array satisfy a condition. Returns true only if the query evaluates to true for every element. Returns false for empty arrays. #### [](#parameters-44)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A test query to apply to each element. | #### [](#examples-57)Examples ```bloblang root.all_over_21 = this.patrons.all(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"all_over_21":false} # In: {"patrons":[{"id":"1","age":45},{"id":"2","age":23}]} # Out: {"all_over_21":true} ``` ```bloblang root.all_positive = this.values.all(v -> v > 0) # In: {"values":[1,2,3,4,5]} # Out: {"all_positive":true} # In: {"values":[1,-2,3,4,5]} # Out: {"all_positive":false} ``` ### [](#any)any Tests whether at least one element in an array satisfies a condition. Returns true if the query evaluates to true for any element. Returns false for empty arrays. #### [](#parameters-45)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A test query to apply to each element. | #### [](#examples-58)Examples ```bloblang root.any_over_21 = this.patrons.any(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"any_over_21":true} # In: {"patrons":[{"id":"1","age":10},{"id":"2","age":12}]} # Out: {"any_over_21":false} ``` ```bloblang root.has_errors = this.results.any(r -> r.status == "error") # In: {"results":[{"status":"ok"},{"status":"error"},{"status":"ok"}]} # Out: {"has_errors":true} # In: {"results":[{"status":"ok"},{"status":"ok"}]} # Out: {"has_errors":false} ``` ### [](#append)append Adds one or more elements to the end of an array and returns the new array. The original array is not modified. #### [](#parameters-46)Parameters This method accepts any number of arguments. #### [](#examples-59)Examples ```bloblang root.foo = this.foo.append("and", "this") # In: {"foo":["bar","baz"]} # Out: {"foo":["bar","baz","and","this"]} ``` ```bloblang root.combined = this.items.append(this.new_item) # In: {"items":["apple","banana"],"new_item":"orange"} # Out: {"combined":["apple","banana","orange"]} ``` ### [](#assign)assign Merges two objects or arrays with override behavior. For objects, source values replace destination values on key conflicts. Arrays are concatenated. To preserve both values on conflict, use the merge method instead. #### [](#parameters-47)Parameters | Name | Type | Description | | --- | --- | --- | | with | unknown | A value to merge the target value with. | #### [](#examples-60)Examples ```bloblang root = this.foo.assign(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":"foos","second_name":"barer"} ``` Override defaults with user settings: ```bloblang root.config = this.defaults.assign(this.user_settings) # In: {"defaults":{"timeout":30,"retries":3},"user_settings":{"timeout":60}} # Out: {"config":{"retries":3,"timeout":60}} ``` ### [](#collapse)collapse Flattens a nested structure into a single-level object with dot-notation keys representing the full path to each value. Empty arrays and objects are excluded by default. #### [](#parameters-48)Parameters | Name | Type | Description | | --- | --- | --- | | include_empty (optional) | bool | Whether to include empty objects and arrays in the resulting object. Default: false. | #### [](#examples-61)Examples ```bloblang root.result = this.collapse() # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.2.bar":"2"}} ``` Set include\_empty to true to preserve empty objects and arrays in the output: ```bloblang root.result = this.collapse(include_empty: true) # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.1.bar":{},"foo.2.bar":"2","foo.3.bar":[]}} ``` ### [](#concat)concat Concatenates an array value with one or more argument arrays. Introduced in version 4.12.0. #### [](#parameters-49)Parameters This method accepts any number of arguments. #### [](#examples-62)Examples ```bloblang root.foo = this.foo.concat(this.bar, this.baz) # In: {"foo":["a","b"],"bar":["c"],"baz":["d","e","f"]} # Out: {"foo":["a","b","c","d","e","f"]} ``` ### [](#contains)contains Checks whether an array contains an element matching the argument, or an object contains a value matching the argument, and returns a boolean result. Numerical comparisons are made irrespective of the representation type (float versus integer). #### [](#parameters-50)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to test against elements of the target. | #### [](#examples-63)Examples ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} ``` ```bloblang root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} ``` ### [](#diff)diff Compares the current value with another value and returns a detailed changelog describing all differences. The changelog contains operations (create, update, delete) with their paths and values, enabling you to track changes between data versions, implement audit logs, or synchronize data between systems. Introduced in version 4.25.0. #### [](#parameters-51)Parameters | Name | Type | Description | | --- | --- | --- | | other | unknown | The value to compare against the current value. Can be any structured data (object or array). | #### [](#examples-64)Examples Compare two objects to track field changes: ```bloblang root.changes = this.before.diff(this.after) # In: {"before":{"name":"Alice","age":30},"after":{"name":"Alice","age":31,"city":"NYC"}} # Out: {"changes":[{"From":30,"Path":["age"],"To":31,"Type":"update"},{"From":null,"Path":["city"],"To":"NYC","Type":"create"}]} ``` Detect deletions in configuration changes: ```bloblang root.changelog = this.old_config.diff(this.new_config) # In: {"old_config":{"debug":true,"timeout":30},"new_config":{"timeout":60}} # Out: {"changelog":[{"From":true,"Path":["debug"],"To":null,"Type":"delete"},{"From":30,"Path":["timeout"],"To":60,"Type":"update"}]} ``` ### [](#enumerated)enumerated Transforms an array into an array of objects with index and value fields, making it easy to access both the position and content of each element. #### [](#examples-65)Examples ```bloblang root.foo = this.foo.enumerated() # In: {"foo":["bar","baz"]} # Out: {"foo":[{"index":0,"value":"bar"},{"index":1,"value":"baz"}]} ``` Useful for filtering by index position: ```bloblang root.first_two = this.items.enumerated().filter(item -> item.index < 2).map_each(item -> item.value) # In: {"items":["a","b","c","d"]} # Out: {"first_two":["a","b"]} ``` ### [](#exists)exists Checks whether a field exists at the specified dot path within an object. Returns true if the field is present (even if null), false otherwise. #### [](#parameters-52)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path to a field. | #### [](#examples-66)Examples ```bloblang root.result = this.foo.exists("bar.baz") # In: {"foo":{"bar":{"baz":"yep, I exist"}}} # Out: {"result":true} # In: {"foo":{"bar":{}}} # Out: {"result":false} # In: {"foo":{}} # Out: {"result":false} ``` Also returns true for null values if the field exists: ```bloblang root.has_field = this.data.exists("optional_field") # In: {"data":{"optional_field":null}} # Out: {"has_field":true} # In: {"data":{}} # Out: {"has_field":false} ``` ### [](#explode)explode Expands a nested array or object field into multiple documents, distributing elements while preserving the surrounding structure. Useful for denormalizing data. #### [](#parameters-53)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path to a field to explode. | #### [](#examples-67)Examples ##### [](#on-arrays)On arrays When exploding an array, each element becomes a separate document with the array element replacing the original field: ```bloblang root = this.explode("value") # In: {"id":1,"value":["foo","bar","baz"]} # Out: [{"id":1,"value":"foo"},{"id":1,"value":"bar"},{"id":1,"value":"baz"}] ``` ##### [](#on-objects)On objects When exploding an object, the output keys match the nested object’s keys, with values being the full document where the target field is replaced by each nested value: ```bloblang root = this.explode("value") # In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}} # Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}} ``` ### [](#filter)filter Returns a new array or object containing only elements that satisfy the provided condition. Elements for which the query returns true are kept, all others are removed. #### [](#parameters-54)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A query to apply to each element, if this query resolves to any value other than a boolean true the element will be removed from the result. | #### [](#examples-68)Examples ```bloblang root.new_nums = this.nums.filter(num -> num > 10) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[11,17]} ``` ##### [](#on-objects-2)On objects When filtering objects, the query receives a context with `key` and `value` fields for each entry: ```bloblang root.new_dict = this.dict.filter(item -> item.value.contains("foo")) # In: {"dict":{"first":"hello foo","second":"world","third":"this foo is great"}} # Out: {"new_dict":{"first":"hello foo","third":"this foo is great"}} ``` ### [](#find)find > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Searches an array for a matching value and returns the index of the first occurrence. Returns -1 if no match is found. Numeric types are compared by value regardless of representation. #### [](#parameters-55)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to find. | #### [](#examples-69)Examples ```bloblang root.index = this.find("bar") # In: ["foo", "bar", "baz"] # Out: {"index":1} ``` ```bloblang root.index = this.things.find(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz"]} # Out: {"index":1} ``` ### [](#find_all)find_all > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Searches an array for all occurrences of a value and returns an array of matching indexes. Returns an empty array if no matches are found. Numeric types are compared by value regardless of representation. #### [](#parameters-56)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to find. | #### [](#examples-70)Examples ```bloblang root.index = this.find_all("bar") # In: ["foo", "bar", "baz", "bar"] # Out: {"index":[1,3]} ``` ```bloblang root.indexes = this.things.find_all(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz", "bar", "buz"]} # Out: {"indexes":[1,3]} ``` ### [](#find_all_by)find_all_by > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Searches an array for all elements that satisfy a condition and returns an array of their indexes. Returns an empty array if no elements match. #### [](#parameters-57)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute for each element. | #### [](#examples-71)Examples ```bloblang root.index = this.find_all_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":[0,2]} ``` Find all indexes matching criteria: ```bloblang root.error_indexes = this.logs.find_all_by(log -> log.level == "error") # In: {"logs":[{"level":"info"},{"level":"error"},{"level":"warn"},{"level":"error"}]} # Out: {"error_indexes":[1,3]} ``` ### [](#find_by)find_by > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Searches an array for the first element that satisfies a condition and returns its index. Returns -1 if no element matches the query. #### [](#parameters-58)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute for each element. | #### [](#examples-72)Examples ```bloblang root.index = this.find_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":0} ``` Find first object matching criteria: ```bloblang root.first_adult = this.users.find_by(u -> u.age >= 18) # In: {"users":[{"name":"Alice","age":15},{"name":"Bob","age":22},{"name":"Carol","age":19}]} # Out: {"first_adult":1} ``` ### [](#flatten)flatten Flattens an array by one level, expanding nested arrays into the parent array. Only the first level of nesting is removed. #### [](#examples-73)Examples ```bloblang root.result = this.flatten() # In: ["foo",["bar","baz"],"buz"] # Out: {"result":["foo","bar","baz","buz"]} ``` Deeper nesting requires multiple flatten calls: ```bloblang root.result = this.data.flatten() # In: {"data":["a",["b",["c","d"]],"e"]} # Out: {"result":["a","b",["c","d"],"e"]} ``` ### [](#fold)fold Reduces an array to a single value by iteratively applying a function. Also known as reduce or aggregate. The query receives an accumulator (tally) and current element (value) for each iteration. #### [](#parameters-59)Parameters | Name | Type | Description | | --- | --- | --- | | initial | unknown | The initial value to start the fold with. For example, an empty object {}, a zero count 0, or an empty string "". | | query | query expression | A query to apply for each element. The query is provided an object with two fields; tally containing the current tally, and value containing the value of the current element. The query should result in a new tally to be passed to the next element query. | #### [](#examples-74)Examples Sum numbers in an array: ```bloblang root.sum = this.foo.fold(0, item -> item.tally + item.value) # In: {"foo":[3,8,11]} # Out: {"sum":22} ``` Concatenate strings: ```bloblang root.result = this.foo.fold("", item -> "%v%v".format(item.tally, item.value)) # In: {"foo":["hello ", "world"]} # Out: {"result":"hello world"} ``` Merge an array of objects into a single object: ```bloblang root.smoothie = this.fruits.fold({}, item -> item.tally.merge(item.value)) # In: {"fruits":[{"apple":5},{"banana":3},{"orange":8}]} # Out: {"smoothie":{"apple":5,"banana":3,"orange":8}} ``` ### [](#get)get Extract a field value, identified via a [dot path](https://docs.redpanda.com/connect/configuration/field_paths/), from an object. #### [](#parameters-60)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path identifying a field to obtain. | #### [](#examples-75)Examples ```bloblang root.result = this.foo.get(this.target) # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"bar"} # Out: {"result":"from bar"} # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"baz"} # Out: {"result":"from baz"} ``` ### [](#index)index Extract an element from an array by an index. The index can be negative, and if so the element will be selected from the end counting backwards starting from -1. E.g. an index of -1 returns the last element, an index of -2 returns the element before the last, and so on. #### [](#parameters-61)Parameters | Name | Type | Description | | --- | --- | --- | | index | integer | The index to obtain from an array. | #### [](#examples-76)Examples ```bloblang root.last_name = this.names.index(-1) # In: {"names":["rachel","stevens"]} # Out: {"last_name":"stevens"} ``` It is also possible to use this method on byte arrays, in which case the selected element will be returned as an integer: ```bloblang root.last_byte = this.name.bytes().index(-1) # In: {"name":"foobar bazson"} # Out: {"last_byte":110} ``` ### [](#join)join Concatenates an array of strings into a single string with an optional delimiter between elements. Use for building CSV strings, URLs, or combining text fragments. #### [](#parameters-62)Parameters | Name | Type | Description | | --- | --- | --- | | delimiter (optional) | string | An optional delimiter to add between each string. | #### [](#examples-77)Examples ```bloblang root.joined_words = this.words.join() root.joined_numbers = this.numbers.map_each(this.string()).join(",") # In: {"words":["hello","world"],"numbers":[3,8,11]} # Out: {"joined_numbers":"3,8,11","joined_words":"helloworld"} ``` ### [](#json_path)json_path Executes the given JSONPath expression on an object or array and returns the result. The JSONPath expression syntax can be found at [https://goessner.net/articles/JsonPath/](https://goessner.net/articles/JsonPath/). For more complex logic, you can use Gval expressions ([https://github.com/PaesslerAG/gval](https://github.com/PaesslerAG/gval)). Introduced in version 4.6.0. #### [](#parameters-63)Parameters | Name | Type | Description | | --- | --- | --- | | expression | string | The JSONPath expression to execute. | #### [](#examples-78)Examples ```bloblang root.all_names = this.json_path("$..name") # In: {"name":"alice","foo":{"name":"bob"}} # Out: {"all_names":["alice","bob"]} # In: {"thing":["this","bar",{"name":"alice"}]} # Out: {"all_names":["alice"]} ``` ```bloblang root.text_objects = this.json_path("$.body[?(@.type=='text')]") # In: {"body":[{"type":"image","id":"foo"},{"type":"text","id":"bar"}]} # Out: {"text_objects":[{"id":"bar","type":"text"}]} ``` ### [](#json_schema)json_schema > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Checks a [JSON schema](https://json-schema.org/) against a value and returns the value if it matches or throws an error if it does not. To load the schema from a file, pass it the result of the `file` function, which isn’t available in Redpanda Cloud. #### [](#parameters-64)Parameters | Name | Type | Description | | --- | --- | --- | | schema | string | The schema to check values against. | #### [](#examples-79)Examples ```bloblang root = this.json_schema("""{ "type":"object", "properties":{ "foo":{ "type":"string" } } }""") # In: {"foo":"bar"} # Out: {"foo":"bar"} # In: {"foo":5} # Out: Error("failed assignment (line 1): field `this`: foo invalid type. expected: string, given: integer") ``` ### [](#key_values)key_values Converts an object into an array of key-value pair objects. Each element has a 'key' field and a 'value' field. Order is not guaranteed unless sorted. #### [](#examples-80)Examples ```bloblang root.foo_key_values = this.foo.key_values().sort_by(pair -> pair.key) # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_key_values":[{"key":"bar","value":1},{"key":"baz","value":2}]} ``` Filter object entries by value: ```bloblang root.large_items = this.items.key_values().filter(pair -> pair.value > 15).map_each(pair -> pair.key) # In: {"items":{"a":5,"b":15,"c":20,"d":3}} # Out: {"large_items":["c"]} ``` ### [](#keys)keys Extracts all keys from an object and returns them as a sorted array. #### [](#examples-81)Examples ```bloblang root.foo_keys = this.foo.keys() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_keys":["bar","baz"]} ``` Check if specific keys exist: ```bloblang root.has_id = this.data.keys().contains("id") # In: {"data":{"id":123,"name":"test"}} # Out: {"has_id":true} ``` ### [](#length)length Returns the size of an array (element count) or object (key count). #### [](#examples-82)Examples ```bloblang root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} ``` ### [](#map_each)map_each Applies a mapping query to each element of an array or each value in an object. Returns a new collection with the transformed values. #### [](#parameters-65)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query that will be used to map each element. | #### [](#examples-83)Examples ##### [](#on-arrays-2)On arrays Transforms each array element using a query. Return deleted() to remove an element, or the new value to replace it: ```bloblang root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} ``` ##### [](#on-objects-3)On objects Transforms each object value using a query. The query receives an object with 'key' and 'value' fields for each entry: ```bloblang root.new_dict = this.dict.map_each(item -> item.value.uppercase()) # In: {"dict":{"foo":"hello","bar":"world"}} # Out: {"new_dict":{"bar":"WORLD","foo":"HELLO"}} ``` ### [](#map_each_key)map_each_key Transforms object keys using a query. The query receives each key as a string and must return a new string key. Use this to rename or transform keys while preserving values. #### [](#parameters-66)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query that will be used to map each key. | #### [](#examples-84)Examples ```bloblang root.new_dict = this.dict.map_each_key(key -> key.uppercase()) # In: {"dict":{"keya":"hello","keyb":"world"}} # Out: {"new_dict":{"KEYA":"hello","KEYB":"world"}} ``` Conditionally transform keys: ```bloblang root = this.map_each_key(key -> if key.contains("kafka") { "_" + key }) # In: {"amqp_key":"foo","kafka_key":"bar","kafka_topic":"baz"} # Out: {"_kafka_key":"bar","_kafka_topic":"baz","amqp_key":"foo"} ``` ### [](#merge)merge Combines two objects or arrays. When merging objects, conflicting keys create arrays containing both values. Arrays are concatenated. For key override behavior instead, use the assign method. #### [](#parameters-67)Parameters | Name | Type | Description | | --- | --- | --- | | with | unknown | A value to merge the target value with. | #### [](#examples-85)Examples ```bloblang root = this.foo.merge(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":["bars","foos"],"second_name":"barer"} ``` Merge arrays: ```bloblang root.combined = this.list1.merge(this.list2) # In: {"list1":["a","b"],"list2":["c","d"]} # Out: {"combined":["a","b","c","d"]} ``` ### [](#patch)patch Applies a changelog (created by the diff method) to the current value, transforming it according to the specified operations. This enables you to synchronize data, replay changes, or implement event sourcing patterns by applying recorded changes to reconstruct state. Introduced in version 4.25.0. #### [](#parameters-68)Parameters | Name | Type | Description | | --- | --- | --- | | changelog | unknown | The changelog array to apply. Should be in the format returned by the diff method, containing Type, Path, From, and To fields for each change. | #### [](#examples-86)Examples Apply recorded changes to update an object: ```bloblang root.updated = this.current.patch(this.changelog) # In: {"current":{"name":"Alice","age":30},"changelog":[{"Type":"update","Path":["age"],"From":30,"To":31},{"Type":"create","Path":["city"],"From":null,"To":"NYC"}]} # Out: {"updated":{"age":31,"city":"NYC","name":"Alice"}} ``` Restore previous state by applying inverse changes: ```bloblang root.restored = this.modified.patch(this.reverse_changelog) # In: {"modified":{"timeout":60},"reverse_changelog":[{"Type":"create","Path":["debug"],"From":null,"To":true},{"Type":"update","Path":["timeout"],"From":60,"To":30}]} # Out: {"restored":{"debug":true,"timeout":30}} ``` ### [](#slice)slice Extract a slice from an array by specifying two indices, a low and high bound, which selects a half-open range that includes the first element, but excludes the last one. If the second index is omitted then it defaults to the length of the input sequence. #### [](#parameters-69)Parameters | Name | Type | Description | | --- | --- | --- | | low | integer | The low bound, which is the first element of the selection, or if negative selects from the end. | | high (optional) | integer | An optional high bound. | #### [](#examples-87)Examples ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} ``` ### [](#sort)sort Sorts an array in ascending order. Works with strings and numbers. For custom sorting logic, provide a comparison query that receives 'left' and 'right' elements. #### [](#parameters-70)Parameters | Name | Type | Description | | --- | --- | --- | | compare (optional) | query expression | An optional query that should explicitly compare elements left and right and provide a boolean result. | #### [](#examples-88)Examples ```bloblang root.sorted = this.foo.sort() # In: {"foo":["bbb","ccc","aaa"]} # Out: {"sorted":["aaa","bbb","ccc"]} ``` Custom comparison for complex objects - return true if left < right: ```bloblang root.sorted = this.foo.sort(item -> item.left.v < item.right.v) # In: {"foo":[{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"},{"id":"baz","v":"aaa"}]} # Out: {"sorted":[{"id":"baz","v":"aaa"},{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"}]} ``` ### [](#sort_by)sort_by Sorts an array by a value extracted from each element using a query. The extracted values determine sort order and must all be strings or numbers. #### [](#parameters-71)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to apply to each element that yields a value used for sorting. | #### [](#examples-89)Examples ```bloblang root.sorted = this.foo.sort_by(ele -> ele.id) # In: {"foo":[{"id":"bbb","message":"bar"},{"id":"aaa","message":"foo"},{"id":"ccc","message":"baz"}]} # Out: {"sorted":[{"id":"aaa","message":"foo"},{"id":"bbb","message":"bar"},{"id":"ccc","message":"baz"}]} ``` Sort by numeric field: ```bloblang root.sorted = this.items.sort_by(item -> item.priority) # In: {"items":[{"name":"low","priority":3},{"name":"high","priority":1},{"name":"med","priority":2}]} # Out: {"sorted":[{"name":"high","priority":1},{"name":"med","priority":2},{"name":"low","priority":3}]} ``` ### [](#squash)squash Squashes an array of objects into a single object, where key collisions result in the values being merged (following similar rules as the `.merge()` method). Introduced in version 4.6.0. #### [](#examples-90)Examples ```bloblang root.locations = this.locations.map_each(loc -> {loc.state: [loc.name]}).squash() # In: {"locations":[{"name":"Seattle","state":"WA"},{"name":"New York","state":"NY"},{"name":"Bellevue","state":"WA"},{"name":"Olympia","state":"WA"}]} # Out: {"locations":{"NY":["New York"],"WA":["Seattle","Bellevue","Olympia"]}} ``` ### [](#sum)sum Calculates the sum of all numeric values in an array. Non-numeric values cause an error. #### [](#examples-91)Examples ```bloblang root.sum = this.foo.sum() # In: {"foo":[3,8,4]} # Out: {"sum":15} ``` Works with decimals: ```bloblang root.total = this.prices.sum() # In: {"prices":[10.5,20.25,5.00]} # Out: {"total":35.75} ``` ### [](#unique)unique Removes duplicate values from an array, keeping the first occurrence of each unique value. Strings and numbers are treated as distinct types ("5" differs from 5). #### [](#parameters-72)Parameters | Name | Type | Description | | --- | --- | --- | | emit (optional) | query expression | An optional query that can be used in order to yield a value for each element to determine uniqueness. | #### [](#examples-92)Examples ```bloblang root.uniques = this.foo.unique() # In: {"foo":["a","b","a","c"]} # Out: {"uniques":["a","b","c"]} ``` Use a query to determine uniqueness by a field: ```bloblang root.unique_users = this.users.unique(u -> u.id) # In: {"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":1,"name":"Alice Duplicate"}]} # Out: {"unique_users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]} ``` ### [](#values)values Extracts all values from an object and returns them as an array. Order is not guaranteed unless the result is sorted. #### [](#examples-93)Examples ```bloblang root.foo_vals = this.foo.values().sort() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_vals":[1,2]} ``` Find max value in object: ```bloblang root.max = this.scores.values().sort().index(-1) # In: {"scores":{"player1":85,"player2":92,"player3":78}} # Out: {"max":92} ``` ### [](#with)with Returns an object where all but one or more [field path](https://docs.redpanda.com/connect/configuration/field_paths/) arguments are removed. Each path specifies a specific field to be retained from the input object, allowing for nested fields. If a key within a nested path does not exist then it is ignored. Introduced in version 4.10.0. #### [](#parameters-73)Parameters This method accepts any number of arguments. #### [](#examples-94)Examples ```bloblang root = this.with("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"d":"fourth","inner":{"a":"first","c":"third"}} ``` ### [](#without)without Removes specified fields from an object using dot-notation paths. Returns a new object with the fields removed. Non-existent paths are safely ignored. #### [](#parameters-74)Parameters This method accepts any number of arguments. #### [](#examples-95)Examples ```bloblang root = this.without("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"e":"fifth","inner":{"b":"second"}} ``` Remove sensitive fields: ```bloblang root = this.without("password","ssn","creditCard") # In: {"username":"alice","password":"secret","email":"alice@example.com","ssn":"123-45-6789"} # Out: {"email":"alice@example.com","username":"alice"} ``` ### [](#zip)zip Zip an array value with one or more argument arrays. Each array must match in length. Introduced in version 4.19.0. #### [](#parameters-75)Parameters This method accepts any number of arguments. #### [](#examples-96)Examples ```bloblang root.foo = this.foo.zip(this.bar, this.baz) # In: {"foo":["a","b","c"],"bar":[1,2,3],"baz":[4,5,6]} # Out: {"foo":[["a",1,4],["b",2,5],["c",3,6]]} ``` ## [](#parsing)Parsing ### [](#bloblang)bloblang > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Executes an argument Bloblang mapping on the target. This method can be used in order to execute dynamic mappings. Imports and functions that interact with the environment, such as `file` and `env`, or that access message information directly, such as `content` or `json`, are not enabled for dynamic Bloblang mappings. #### [](#parameters-76)Parameters | Name | Type | Description | | --- | --- | --- | | mapping | string | The mapping to execute. | #### [](#examples-97)Examples ```bloblang root.body = this.body.bloblang(this.mapping) # In: {"body":{"foo":"hello world"},"mapping":"root.foo = this.foo.uppercase()"} # Out: {"body":{"foo":"HELLO WORLD"}} # In: {"body":{"foo":"hello world 2"},"mapping":"root.foo = this.foo.capitalize()"} # Out: {"body":{"foo":"Hello World 2"}} ``` ### [](#format_json)format_json > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Serializes a target value into a pretty-printed JSON byte array (with 4 space indentation by default). #### [](#parameters-77)Parameters | Name | Type | Description | | --- | --- | --- | | indent (optional) | string | Indentation string. Each element in a JSON object or array will begin on a new, indented line followed by one or more copies of indent according to the indentation nesting. Default: " ". | | no_indent (optional) | bool | Disable indentation. Default: false. | | escape_html (optional) | bool | Escape problematic HTML characters. Default: true. | #### [](#examples-98)Examples ```bloblang root = this.doc.format_json() # In: {"doc":{"foo":"bar"}} # Out: { "foo": "bar" } ``` Pass a string to the `indent` parameter in order to customise the indentation: ```bloblang root = this.format_json(" ") # In: {"doc":{"foo":"bar"}} # Out: { "doc": { "foo": "bar" } } ``` Use the `.string()` method in order to coerce the result into a string: ```bloblang root.doc = this.doc.format_json().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"{\n \"foo\": \"bar\"\n}"} ``` Set the `no_indent` parameter to true to disable indentation. The result is equivalent to calling `bytes()`: ```bloblang root = this.doc.format_json(no_indent: true) # In: {"doc":{"foo":"bar"}} # Out: {"foo":"bar"} ``` Escapes problematic HTML characters: ```bloblang root = this.doc.format_json() # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo\u0026bar@benthos.dev", "name": "foo\u003ebar" } ``` Set the `escape_html` parameter to false to disable escaping of problematic HTML characters: ```bloblang root = this.doc.format_json(escape_html: false) # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo&bar@benthos.dev", "name": "foo>bar" } ``` ### [](#format_msgpack)format_msgpack Serializes structured data into MessagePack binary format. MessagePack is a compact binary serialization that is faster and more space-efficient than JSON, making it ideal for network transmission and storage of structured data. Returns a byte array that can be further encoded as needed. #### [](#examples-99)Examples Serialize object to MessagePack and encode as hex for transmission: ```bloblang root = this.format_msgpack().encode("hex") # In: {"foo":"bar"} # Out: 81a3666f6fa3626172 ``` Serialize data to MessagePack and base64 encode for embedding in JSON: ```bloblang root.msgpack_payload = this.data.format_msgpack().encode("base64") # In: {"data":{"foo":"bar"}} # Out: {"msgpack_payload":"gaNmb2+jYmFy"} ``` ### [](#format_xml)format_xml Serializes an object into an XML document. Converts structured data to XML format with support for attributes (prefixed with hyphen), custom indentation, and configurable root element. Returns XML as a byte array. Introduced in version 4.11.0. #### [](#parameters-78)Parameters | Name | Type | Description | | --- | --- | --- | | indent (optional) | string | String to use for each level of indentation (default is 4 spaces). Each nested XML element will be indented by this string. Default: " ". | | no_indent (optional) | bool | Disable indentation and newlines to produce compact XML on a single line. Default: false. | | root_tag (optional) | string | Custom name for the root XML element. By default, the root element name is derived from the first key in the object. | #### [](#examples-100)Examples Serialize object to pretty-printed XML with default indentation: ```bloblang root = this.format_xml() # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: foo bar baz ``` Create compact XML without indentation for smaller message size: ```bloblang root = this.format_xml(no_indent: true) # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: foo bar baz ``` ### [](#format_yaml)format_yaml Serializes a target value into a YAML byte array. #### [](#examples-101)Examples ```bloblang root = this.doc.format_yaml() # In: {"doc":{"foo":"bar"}} # Out: foo: bar ``` Use the `.string()` method in order to coerce the result into a string: ```bloblang root.doc = this.doc.format_yaml().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"foo: bar\n"} ``` ### [](#infer_schema)infer_schema Attempt to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods. Introduced in version 4.62.0. ### [](#parse_csv)parse_csv Attempts to parse a string into an array of objects by following the CSV format described in RFC 4180. #### [](#parameters-79)Parameters | Name | Type | Description | | --- | --- | --- | | parse_header_row (optional) | bool | Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, the output will be an array of row arrays. Default: true. | | delimiter (optional) | string | The delimiter to use for splitting values in each record. It must be a single character. Default: ",". | | lazy_quotes (optional) | bool | If set to true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. Default: false. | #### [](#examples-102)Examples Parses CSV data with a header row: ```bloblang root.orders = this.orders.parse_csv() # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} ``` Parses CSV data without a header row: ```bloblang root.orders = this.orders.parse_csv(false) # In: {"orders":"foo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[["foo 1","bar 1"],["foo 2","bar 2"]]} ``` Parses CSV data delimited by dots: ```bloblang root.orders = this.orders.parse_csv(delimiter:".") # In: {"orders":"foo.bar\nfoo 1.bar 1\nfoo 2.bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} ``` Parses CSV data containing a quote in an unquoted field: ```bloblang root.orders = this.orders.parse_csv(lazy_quotes:true) # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo\" \"2,bar\" \"2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar\" \"2","foo":"foo\" \"2"}]} ``` ### [](#parse_form_url_encoded)parse_form_url_encoded Attempts to parse a url-encoded query string (from an x-www-form-urlencoded request body) and returns a structured result. Introduced in version 4.5.0. #### [](#examples-103)Examples ```bloblang root.values = this.body.parse_form_url_encoded() # In: {"body":"noise=meow&animal=cat&fur=orange&fur=fluffy"} # Out: {"values":{"animal":"cat","fur":["orange","fluffy"],"noise":"meow"}} ``` ### [](#parse_json)parse_json Attempts to parse a string as a JSON document and returns the result. #### [](#parameters-80)Parameters | Name | Type | Description | | --- | --- | --- | | use_number (optional) | bool | An optional flag that when set makes parsing numbers as json.Number instead of the default float64. | #### [](#examples-104)Examples ```bloblang root.doc = this.doc.parse_json() # In: {"doc":"{\"foo\":\"bar\"}"} # Out: {"doc":{"foo":"bar"}} ``` ```bloblang root.doc = this.doc.parse_json(use_number: true) # In: {"doc":"{\"foo\":\"11380878173205700000000000000000000000000000000\"}"} # Out: {"doc":{"foo":"11380878173205700000000000000000000000000000000"}} ``` ### [](#parse_logfmt)parse_logfmt Attempts to parse a logfmt encoded string into an object. A logfmt string contains key=value pairs separated by spaces, where values can optionally be quoted. #### [](#examples-105)Examples ```bloblang root = this.msg.parse_logfmt() # In: {"msg":"level=info msg=\"hello world\" dur=1.5s"} # Out: {"dur":"1.5s","level":"info","msg":"hello world"} ``` ### [](#parse_msgpack)parse_msgpack Parses MessagePack binary data into a structured object. MessagePack is an efficient binary serialization format that is more compact than JSON while maintaining similar data structures. Commonly used for high-performance APIs and data interchange between microservices. #### [](#examples-106)Examples Parse MessagePack data from hex-encoded content: ```bloblang root = content().decode("hex").parse_msgpack() # In: 81a3666f6fa3626172 # Out: {"foo":"bar"} ``` Parse MessagePack from base64-encoded field: ```bloblang root.decoded = this.msgpack_data.decode("base64").parse_msgpack() # In: {"msgpack_data":"gaNmb2+jYmFy"} # Out: {"decoded":{"foo":"bar"}} ``` ### [](#parse_parquet)parse_parquet Parses Apache Parquet binary data into an array of objects. Parquet is a columnar storage format optimized for analytics, commonly used with big data systems like Apache Spark, Hive, and cloud data warehouses. Each row in the Parquet file becomes an object in the output array. Introduced in version 4.4.0. #### [](#parameters-81)Parameters | Name | Type | Description | | --- | --- | --- | | byte_array_as_string (optional) | bool | Deprecated: This parameter is no longer used. Default: false. | #### [](#examples-107)Examples Parse Parquet file data into structured objects: ```bloblang root.records = content().parse_parquet() ``` Process Parquet data from a field and extract specific columns: ```bloblang root.users = this.parquet_data.parse_parquet().map_each(row -> {"name": row.name, "email": row.email}) ``` ### [](#parse_url)parse_url Attempts to parse a URL from a string value, returning a structured result that describes the various facets of the URL. The fields returned within the structured result roughly follow [https://pkg.go.dev/net/url#URL](https://pkg.go.dev/net/url#URL), and may be expanded in future in order to present more information. #### [](#examples-108)Examples ```bloblang root.foo_url = this.foo_url.parse_url() # In: {"foo_url":"https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/"} # Out: {"foo_url":{"fragment":"","host":"docs.redpanda.com","opaque":"","path":"/redpanda-connect/guides/bloblang/about/","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}} ``` ```bloblang root.username = this.url.parse_url().user.name | "unknown" # In: {"url":"amqp://foo:bar@127.0.0.1:5672/"} # Out: {"username":"foo"} # In: {"url":"redis://localhost:6379"} # Out: {"username":"unknown"} ``` ### [](#parse_xml)parse_xml Parses an XML document into a structured object. Converts XML elements to JSON-like objects following these rules: - Element attributes are prefixed with a hyphen (e.g., `-id` for an `id` attribute) - Elements with both attributes and text content store the text in a `#text` field - Repeated elements become arrays - XML comments, directives, and processing instructions are ignored - Optionally cast numeric and boolean strings to their proper types. #### [](#parameters-82)Parameters | Name | Type | Description | | --- | --- | --- | | cast (optional) | bool | Whether to automatically cast numeric and boolean string values to their proper types. When false, all values remain as strings. Default: false. | #### [](#examples-109)Examples Parse XML document into object structure: ```bloblang root.doc = this.doc.parse_xml() # In: {"doc":"This is a titleThis is some content"} # Out: {"doc":{"root":{"content":"This is some content","title":"This is a title"}}} ``` Parse XML with type casting enabled to convert strings to numbers and booleans: ```bloblang root.doc = this.doc.parse_xml(cast: true) # In: {"doc":"This is a title123True"} # Out: {"doc":{"root":{"bool":true,"number":{"#text":123,"-id":99},"title":"This is a title"}}} ``` ### [](#parse_yaml)parse_yaml Attempts to parse a string as a single YAML document and returns the result. #### [](#examples-110)Examples ```bloblang root.doc = this.doc.parse_yaml() # In: {"doc":"foo: bar"} # Out: {"doc":{"foo":"bar"}} ``` ## [](#regular-expressions)Regular expressions ### [](#re_find_all)re_find_all Finds all matches of a regular expression in a string and returns them as an array. Use for extracting multiple patterns or validating repeating structures. #### [](#parameters-83)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-111)Examples ```bloblang root.matches = this.value.re_find_all("a.") # In: {"value":"paranormal"} # Out: {"matches":["ar","an","al"]} ``` ```bloblang root.numbers = this.text.re_find_all("[0-9]+") # In: {"text":"I have 2 apples and 15 oranges"} # Out: {"numbers":["2","15"]} ``` ### [](#re_find_all_object)re_find_all_object Finds all regex matches and returns an array of objects with named capture groups as keys. Each object represents one match with its captured groups. Use for parsing multiple structured records from text. #### [](#parameters-84)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-112)Examples ```bloblang root.matches = this.value.re_find_all_object("a(?Px*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[{"0":"axxb","foo":"xx"},{"0":"ab","foo":""}]} ``` ```bloblang root.matches = this.value.re_find_all_object("(?m)(?P\\w+):\\s+(?P\\w+)$") # In: {"value":"option1: value1\noption2: value2\noption3: value3"} # Out: {"matches":[{"0":"option1: value1","key":"option1","value":"value1"},{"0":"option2: value2","key":"option2","value":"value2"},{"0":"option3: value3","key":"option3","value":"value3"}]} ``` ### [](#re_find_all_submatch)re_find_all_submatch Finds all regex matches and their capture groups, returning an array of arrays where each inner array contains the full match and captured subgroups. Use for extracting structured data with capture groups. #### [](#parameters-85)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-113)Examples ```bloblang root.matches = this.value.re_find_all_submatch("a(x*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[["axxb","xx"],["ab",""]]} ``` ```bloblang root.emails = this.text.re_find_all_submatch("(\\w+)@(\\w+\\.\\w+)") # In: {"text":"Contact: alice@example.com or bob@test.org"} # Out: {"emails":[["alice@example.com","alice","example.com"],["bob@test.org","bob","test.org"]]} ``` ### [](#re_find_object)re_find_object Finds the first regex match and returns an object with named capture groups as keys (or numeric indices for unnamed groups). The key "0" contains the full match. Use for parsing structured text into fields. #### [](#parameters-86)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-114)Examples ```bloblang root.matches = this.value.re_find_object("a(?Px*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":{"0":"axxb","foo":"xx"}} ``` ```bloblang root.matches = this.value.re_find_object("(?P\\w+):\\s+(?P\\w+)") # In: {"value":"option1: value1"} # Out: {"matches":{"0":"option1: value1","key":"option1","value":"value1"}} ``` ### [](#re_match)re_match Tests if a regular expression matches anywhere in a string, returning `true` or `false`. Use for validation or conditional routing based on patterns. #### [](#parameters-87)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-115)Examples ```bloblang root.matches = this.value.re_match("[0-9]") # In: {"value":"there are 10 puppies"} # Out: {"matches":true} # In: {"value":"there are ten puppies"} # Out: {"matches":false} ``` ### [](#re_replace)re_replace Replaces all regex matches with a replacement string that can reference capture groups using `$1`, `$2`, etc. Use for pattern-based transformations or data reformatting. #### [](#parameters-88)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | | value | string | The value to replace with. | ### [](#re_replace_all)re_replace_all Replaces all regex matches with a replacement string that can reference capture groups using `$1`, `$2`, etc. Use for pattern-based transformations or data reformatting. #### [](#parameters-89)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | | value | string | The value to replace with. | #### [](#examples-116)Examples ```bloblang root.new_value = this.value.re_replace_all("ADD ([0-9]+)","+($1)") # In: {"value":"foo ADD 70"} # Out: {"new_value":"foo +(70)"} ``` ```bloblang root.masked = this.email.re_replace_all("(\\w{2})\\w+@", "$1***@") # In: {"email":"alice@example.com"} # Out: {"masked":"al***@example.com"} ``` ## [](#sql)SQL ### [](#vector)vector Converts an array of numbers into a vector type suitable for insertion into SQL databases with vector/embedding support. This is commonly used with PostgreSQL’s pgvector extension for storing and querying machine learning embeddings, enabling similarity search and vector operations in your database. Introduced in version 4.33.0. #### [](#examples-117)Examples Convert embeddings array to vector for pgvector storage: ```bloblang root.embedding = this.embeddings.vector() root.text = this.text ``` Process ML model output into database-ready vector format: ```bloblang root.doc_id = this.id root.vector_embedding = this.model_output.map_each(num -> num.number()).vector() ``` ## [](#string-manipulation)String manipulation ### [](#capitalize)capitalize Converts the first letter of each word in a string to uppercase (title case). Useful for formatting names, titles, and headings. #### [](#examples-118)Examples ```bloblang root.title = this.title.capitalize() # In: {"title":"the foo bar"} # Out: {"title":"The Foo Bar"} ``` ```bloblang root.name = this.name.capitalize() # In: {"name":"alice smith"} # Out: {"name":"Alice Smith"} ``` ### [](#compare_argon2)compare_argon2 Checks whether a string matches a hashed secret using Argon2. Introduced in version 4.12.0. #### [](#parameters-90)Parameters | Name | Type | Description | | --- | --- | --- | | hashed_secret | string | The hashed secret to compare with the input. This must be a fully-qualified string which encodes the Argon2 options used to generate the hash. | #### [](#examples-119)Examples ```bloblang root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} ``` ```bloblang root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} ``` ### [](#compare_bcrypt)compare_bcrypt Checks whether a string matches a hashed secret using bcrypt. Introduced in version 4.12.0. #### [](#parameters-91)Parameters | Name | Type | Description | | --- | --- | --- | | hashed_secret | string | The hashed secret value to compare with the input. | #### [](#examples-120)Examples ```bloblang root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} ``` ```bloblang root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} ``` ### [](#contains-2)contains Checks whether a string contains a substring and returns a boolean result. #### [](#parameters-92)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to test against elements of the target. | #### [](#examples-121)Examples ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} ``` ### [](#escape_html)escape_html Escapes special HTML characters (`<`, `>`, `&`, `'`, `"`) to make a string safe for HTML output. Use when embedding untrusted text in HTML to prevent XSS vulnerabilities. #### [](#examples-122)Examples ```bloblang root.escaped = this.value.escape_html() # In: {"value":"foo & bar"} # Out: {"escaped":"foo & bar"} ``` ```bloblang root.safe_html = this.user_input.escape_html() # In: {"user_input":""} # Out: {"safe_html":"<script>alert('xss')</script>"} ``` ### [](#escape_url_path)escape_url_path Encodes a string for safe use in URL path segments using percent-encoding. Unlike `escape_url_query`, spaces are encoded as `%20` instead of `+`. Use when building URL paths with dynamic segments. #### [](#examples-123)Examples ```bloblang root.escaped = this.value.escape_url_path() # In: {"value":"foo & bar"} # Out: {"escaped":"foo%20&%20bar"} ``` ```bloblang root.url = "https://example.com/docs/" + this.path.escape_url_path() # In: {"path":"my document.pdf"} # Out: {"url":"https://example.com/docs/my%20document.pdf"} ``` ### [](#escape_url_query)escape_url_query Encodes a string for safe use in URL query parameters. Converts spaces to `+` and special characters to percent-encoded values. Use when building URLs with dynamic query parameters. #### [](#examples-124)Examples ```bloblang root.escaped = this.value.escape_url_query() # In: {"value":"foo & bar"} # Out: {"escaped":"foo+%26+bar"} ``` ```bloblang root.url = "https://example.com?search=" + this.query.escape_url_query() # In: {"query":"hello world!"} # Out: {"url":"https://example.com?search=hello+world%21"} ``` ### [](#filepath_join)filepath_join Combines an array of path components into a single OS-specific file path using the correct separator (`/` on Unix, `\` on Windows). Use for constructing file paths from components. #### [](#examples-125)Examples ```bloblang root.path = this.path_elements.filepath_join() # In: {"path_elements":["/foo/","bar.txt"]} # Out: {"path":"/foo/bar.txt"} ``` ### [](#filepath_split)filepath_split Separates a file path into directory and filename components, returning a two-element array `[directory, filename]`. Use for extracting the filename or directory from a full path. #### [](#examples-126)Examples ```bloblang root.path_sep = this.path.filepath_split() # In: {"path":"/foo/bar.txt"} # Out: {"path_sep":["/foo/","bar.txt"]} # In: {"path":"baz.txt"} # Out: {"path_sep":["","baz.txt"]} ``` ### [](#format)format Formats a string using Go’s printf-style formatting with the string as the format template. Supports all Go format verbs (`%s`, `%d`, `%v`, etc.). Use for building formatted strings from dynamic values. #### [](#parameters-93)Parameters This method accepts any number of arguments. #### [](#examples-127)Examples ```bloblang root.foo = "%s(%v): %v".format(this.name, this.age, this.fingers) # In: {"name":"lance","age":37,"fingers":13} # Out: {"foo":"lance(37): 13"} ``` ```bloblang root.message = "User %s has %v points".format(this.username, this.score) # In: {"username":"alice","score":100} # Out: {"message":"User alice has 100 points"} ``` ### [](#has_prefix)has_prefix Tests if a string starts with a specified prefix. Returns `true` if the string begins with the prefix, `false` otherwise. Use for conditional logic based on string patterns. #### [](#parameters-94)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | The string to test. | #### [](#examples-128)Examples ```bloblang root.t1 = this.v1.has_prefix("foo") root.t2 = this.v2.has_prefix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":true,"t2":false} ``` ### [](#has_suffix)has_suffix Tests if a string ends with a specified suffix. Returns `true` if the string ends with the suffix, `false` otherwise. Use for filtering or routing based on file extensions or string patterns. #### [](#parameters-95)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | The string to test. | #### [](#examples-129)Examples ```bloblang root.t1 = this.v1.has_suffix("foo") root.t2 = this.v2.has_suffix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":false,"t2":true} ``` ### [](#index_of)index_of Finds the position of a substring within a string. Returns the zero-based index of the first occurrence, or -1 if not found. Useful for searching and string manipulation. #### [](#parameters-96)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | A string to search for. | #### [](#examples-130)Examples ```bloblang root.index = this.thing.index_of("bar") # In: {"thing":"foobar"} # Out: {"index":3} ``` ```bloblang root.index = content().index_of("meow") # In: the cat meowed, the dog woofed # Out: {"index":8} ``` ### [](#length-2)length Returns the character count of a string. #### [](#examples-131)Examples ```bloblang root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} ``` ### [](#lowercase)lowercase Converts all letters in a string to lowercase. Use for case-insensitive comparisons, normalization, or formatting output. #### [](#examples-132)Examples ```bloblang root.foo = this.foo.lowercase() # In: {"foo":"HELLO WORLD"} # Out: {"foo":"hello world"} ``` ```bloblang root.email = this.user_email.lowercase() # In: {"user_email":"User@Example.COM"} # Out: {"email":"user@example.com"} ``` ### [](#quote)quote Wraps a string in double quotes and escapes special characters (newlines, tabs, etc.) using Go escape sequences. Use for generating string literals or preparing strings for JSON-like formats. #### [](#examples-133)Examples ```bloblang root.quoted = this.thing.quote() # In: {"thing":"foo\nbar"} # Out: {"quoted":"\"foo\\nbar\""} ``` ```bloblang root.literal = this.text.quote() # In: {"text":"hello\tworld"} # Out: {"literal":"\"hello\\tworld\""} ``` ### [](#repeat)repeat Creates a new string by repeating the input string a specified number of times. Use for generating padding, separators, or test data. #### [](#parameters-97)Parameters | Name | Type | Description | | --- | --- | --- | | count | integer | The number of times to repeat the string. | #### [](#examples-134)Examples ```bloblang root.repeated = this.name.repeat(3) root.not_repeated = this.name.repeat(0) # In: {"name":"bob"} # Out: {"not_repeated":"","repeated":"bobbobbob"} ``` ```bloblang root.separator = "-".repeat(10) # In: {} # Out: {"separator":"----------"} ``` ### [](#replace)replace Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings. #### [](#parameters-98)Parameters | Name | Type | Description | | --- | --- | --- | | old | string | A string to match against. | | new | string | A string to replace with. | ### [](#replace_all)replace_all Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings. #### [](#parameters-99)Parameters | Name | Type | Description | | --- | --- | --- | | old | string | A string to match against. | | new | string | A string to replace with. | #### [](#examples-135)Examples ```bloblang root.new_value = this.value.replace_all("foo","dog") # In: {"value":"The foo ate my homework"} # Out: {"new_value":"The dog ate my homework"} ``` ```bloblang root.clean = this.text.replace_all(" ", " ") # In: {"text":"hello world foo"} # Out: {"clean":"hello world foo"} ``` ### [](#replace_all_many)replace_all_many Performs multiple find-and-replace operations in sequence using an array of `[old, new]` pairs. More efficient than chaining multiple `replace_all` calls. Use for bulk text transformations. #### [](#parameters-100)Parameters | Name | Type | Description | | --- | --- | --- | | values | array | An array of values, each even value will be replaced with the following odd value. | #### [](#examples-136)Examples ```bloblang root.new_value = this.value.replace_all_many([ "", "<b>", "", "</b>", "", "<i>", "", "</i>", ]) # In: {"value":"Hello World"} # Out: {"new_value":"<i>Hello</i> <b>World</b>"} ``` ### [](#replace_many)replace_many Performs multiple find-and-replace operations in sequence using an array of `[old, new]` pairs. More efficient than chaining multiple `replace_all` calls. Use for bulk text transformations. #### [](#parameters-101)Parameters | Name | Type | Description | | --- | --- | --- | | values | array | An array of values, each even value will be replaced with the following odd value. | ### [](#reverse)reverse Reverses the order of characters in a string. Unicode-aware for proper handling of multi-byte characters. Use for creating palindrome checks or reversing text data. #### [](#examples-137)Examples ```bloblang root.reversed = this.thing.reverse() # In: {"thing":"backwards"} # Out: {"reversed":"sdrawkcab"} ``` ```bloblang root = content().reverse() # In: {"thing":"backwards"} # Out: }"sdrawkcab":"gniht"{ ``` ### [](#slice-2)slice Extract a slice from a string by specifying two indices, a low and high bound, which selects a half-open range that includes the first character, but excludes the last one. If the second index is omitted then it defaults to the length of the input sequence. #### [](#parameters-102)Parameters | Name | Type | Description | | --- | --- | --- | | low | integer | The low bound, which is the first element of the selection, or if negative selects from the end. | | high (optional) | integer | An optional high bound. | #### [](#examples-138)Examples ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} ``` ### [](#slug)slug Converts a string into a URL-friendly slug by replacing spaces with hyphens, removing special characters, and converting to lowercase. Supports multiple languages for proper transliteration of non-ASCII characters. Introduced in version 4.2.0. #### [](#parameters-103)Parameters | Name | Type | Description | | --- | --- | --- | | lang (optional) | string | Default: "en". | #### [](#examples-139)Examples Create a URL-friendly slug from a string with special characters: ```bloblang root.slug = this.title.slug() # In: {"title":"Hello World! Welcome to Redpanda Connect"} # Out: {"slug":"hello-world-welcome-to-redpanda-connect"} ``` Create a slug preserving French language rules: ```bloblang root.slug = this.title.slug("fr") # In: {"title":"Café & Restaurant"} # Out: {"slug":"cafe-et-restaurant"} ``` ### [](#split)split Splits a string into an array of substrings using a delimiter. Use for parsing CSV-like data, splitting paths, or breaking text into tokens. #### [](#parameters-104)Parameters | Name | Type | Description | | --- | --- | --- | | delimiter | string | The delimiter to split with. | | empty_as_null (optional) | bool | To treat empty substrings as null values. Default: false. | #### [](#examples-140)Examples ```bloblang root.new_value = this.value.split(",") # In: {"value":"foo,bar,baz"} # Out: {"new_value":["foo","bar","baz"]} ``` ```bloblang root.new_value = this.value.split(",", true) # In: {"value":"foo,,qux"} # Out: {"new_value":["foo",null,"qux"]} ``` ```bloblang root.words = this.sentence.split(" ") # In: {"sentence":"hello world from bloblang"} # Out: {"words":["hello","world","from","bloblang"]} ``` ### [](#strip_html)strip_html Removes HTML tags from a string, returning only the text content. Useful for extracting plain text from HTML documents, sanitizing user input, or preparing content for text analysis. Optionally preserves specific HTML elements while stripping all others. #### [](#parameters-105)Parameters | Name | Type | Description | | --- | --- | --- | | preserve (optional) | unknown | Optional array of HTML element names to preserve (e.g., ["strong", "em", "a"]). All other HTML tags will be removed. | #### [](#examples-141)Examples Extract plain text from HTML content: ```bloblang root.plain_text = this.html_content.strip_html() # In: {"html_content":"

Welcome to Redpanda Connect!

"} # Out: {"plain_text":"Welcome to Redpanda Connect!"} ``` Preserve specific HTML elements while removing others: ```bloblang root.sanitized = this.html.strip_html(["strong", "em"]) # In: {"html":"

Some bold and italic text with a

"} # Out: {"sanitized":"Some bold and italic text with a "} ``` ### [](#trim)trim Removes leading and trailing characters from a string. Without arguments, removes whitespace. With a cutset argument, removes any characters in the cutset. Use for cleaning user input or normalizing strings. #### [](#parameters-106)Parameters | Name | Type | Description | | --- | --- | --- | | cutset (optional) | string | An optional string of characters to trim from the target value. | #### [](#examples-142)Examples ```bloblang root.title = this.title.trim("!?") root.description = this.description.trim() # In: {"description":" something happened and its amazing! ","title":"!!!watch out!?"} # Out: {"description":"something happened and its amazing!","title":"watch out"} ``` ### [](#trim_prefix)trim_prefix Removes a specified prefix from the beginning of a string if present. If the string doesn’t start with the prefix, returns the string unchanged. Use for stripping known prefixes from identifiers or paths. Introduced in version 4.12.0. #### [](#parameters-107)Parameters | Name | Type | Description | | --- | --- | --- | | prefix | string | The leading prefix substring to trim from the string. | #### [](#examples-143)Examples ```bloblang root.name = this.name.trim_prefix("foobar_") root.description = this.description.trim_prefix("foobar_") # In: {"description":"unchanged","name":"foobar_blobton"} # Out: {"description":"unchanged","name":"blobton"} ``` ### [](#trim_suffix)trim_suffix Removes a specified suffix from the end of a string if present. If the string doesn’t end with the suffix, returns the string unchanged. Use for stripping file extensions or known suffixes. Introduced in version 4.12.0. #### [](#parameters-108)Parameters | Name | Type | Description | | --- | --- | --- | | suffix | string | The trailing suffix substring to trim from the string. | #### [](#examples-144)Examples ```bloblang root.name = this.name.trim_suffix("_foobar") root.description = this.description.trim_suffix("_foobar") # In: {"description":"unchanged","name":"blobton_foobar"} # Out: {"description":"unchanged","name":"blobton"} ``` ### [](#unescape_html)unescape_html Converts HTML entities back to their original characters. Handles named entities (`&`, `<`), decimal (`á`), and hexadecimal (`&xE1;`) formats. Use for processing HTML content or decoding HTML-escaped data. #### [](#examples-145)Examples ```bloblang root.unescaped = this.value.unescape_html() # In: {"value":"foo & bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.text = this.html.unescape_html() # In: {"html":"<p>Hello & goodbye</p>"} # Out: {"text":"

Hello & goodbye

"} ``` ### [](#unescape_url_path)unescape_url_path Decodes URL path percent-encoding, converting `%20` to spaces and other percent-encoded characters to their original values. Use for parsing URL path segments. #### [](#examples-146)Examples ```bloblang root.unescaped = this.value.unescape_url_path() # In: {"value":"foo%20&%20bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.filename = this.path.unescape_url_path() # In: {"path":"my%20document.pdf"} # Out: {"filename":"my document.pdf"} ``` ### [](#unescape_url_query)unescape_url_query Decodes URL query parameter encoding, converting `+` to spaces and percent-encoded characters to their original values. Use for parsing URL query parameters. #### [](#examples-147)Examples ```bloblang root.unescaped = this.value.unescape_url_query() # In: {"value":"foo+%26+bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.search = this.param.unescape_url_query() # In: {"param":"hello+world%21"} # Out: {"search":"hello world!"} ``` ### [](#unicode_segments)unicode_segments Splits text into segments based on Unicode text segmentation rules. Returns an array of strings representing individual graphemes (visual characters), words (including punctuation and whitespace), or sentences. Handles complex Unicode correctly, including emoji with skin tone modifiers and zero-width joiners. Introduced in version 4.50.0. #### [](#parameters-109)Parameters | Name | Type | Description | | --- | --- | --- | | segmentation_type | string | Type of segmentation: "grapheme", "word", or "sentence" | #### [](#examples-148)Examples Split text into sentences (preserves trailing spaces): ```bloblang root.sentences = this.text.unicode_segments("sentence") # In: {"text":"Hello world. How are you?"} # Out: {"sentences":["Hello world. ","How are you?"]} ``` Split text into grapheme clusters (handles complex emoji correctly): ```bloblang root.graphemes = this.emoji.unicode_segments("grapheme") # In: {"emoji":"👨‍👩‍👧‍👦❤️"} # Out: {"graphemes":["👨‍👩‍👧‍👦","❤️"]} ``` ### [](#unquote)unquote Removes surrounding quotes and interprets escape sequences (`\n`, `\t`, etc.) to their literal characters. Use for parsing quoted string literals. #### [](#examples-149)Examples ```bloblang root.unquoted = this.thing.unquote() # In: {"thing":"\"foo\\nbar\""} # Out: {"unquoted":"foo\nbar"} ``` ```bloblang root.text = this.literal.unquote() # In: {"literal":"\"hello\\tworld\""} # Out: {"text":"hello\tworld"} ``` ### [](#uppercase)uppercase Converts all letters in a string to uppercase. Use for case-insensitive comparisons or formatting output. #### [](#examples-150)Examples ```bloblang root.foo = this.foo.uppercase() # In: {"foo":"hello world"} # Out: {"foo":"HELLO WORLD"} ``` ```bloblang root.code = this.product_code.uppercase() # In: {"product_code":"abc-123"} # Out: {"code":"ABC-123"} ``` ## [](#timestamp-manipulation)Timestamp manipulation ### [](#parse_duration)parse_duration Parses a Go-style duration string into nanoseconds. A duration string is a signed sequence of decimal numbers with unit suffixes like "300ms", "-1.5h", or "2h45m". Valid units: "ns", "us" (or "µs"), "ms", "s", "m", "h". #### [](#examples-151)Examples Parse microseconds to nanoseconds: ```bloblang root.delay_for_ns = this.delay_for.parse_duration() # In: {"delay_for":"50us"} # Out: {"delay_for_ns":50000} ``` Parse hours to seconds: ```bloblang root.delay_for_s = this.delay_for.parse_duration() / 1000000000 # In: {"delay_for":"2h"} # Out: {"delay_for_s":7200} ``` ### [](#parse_duration_iso8601)parse_duration_iso8601 > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Parses an ISO 8601 duration string into nanoseconds. Format: "P\[n\]Y\[n\]M\[n\]DT\[n\]H\[n\]M\[n\]S" or "P\[n\]W". Example: "P3Y6M4DT12H30M5S" means 3 years, 6 months, 4 days, 12 hours, 30 minutes, 5 seconds. Supports fractional seconds with full precision (not just one decimal place). #### [](#examples-152)Examples Parse complex ISO 8601 duration to nanoseconds: ```bloblang root.delay_for_ns = this.delay_for.parse_duration_iso8601() # In: {"delay_for":"P3Y6M4DT12H30M5S"} # Out: {"delay_for_ns":110839937000000000} ``` Parse hours to seconds: ```bloblang root.delay_for_s = this.delay_for.parse_duration_iso8601() / 1000000000 # In: {"delay_for":"PT2H"} # Out: {"delay_for_s":7200} ``` ### [](#ts_add_iso8601)ts_add_iso8601 > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Adds an ISO 8601 duration to a timestamp with calendar-aware precision for years, months, and days. Useful when you need to add durations that account for variable month lengths or leap years. Introduced in version 4.14.0. #### [](#parameters-110)Parameters | Name | Type | Description | | --- | --- | --- | | duration | string | Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) | #### [](#examples-153)Examples Add one year to a timestamp: ```bloblang root.next_year = this.created_at.ts_add_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"next_year":"2021-08-14T05:54:23Z"} ``` Add a complex duration with multiple units: ```bloblang root.future_date = this.created_at.ts_add_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2020-01-01T00:00:00Z"} # Out: {"future_date":"2021-03-04T04:05:06Z"} ``` ### [](#ts_format)ts_format > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts\_strftime for strftime-style formats. Introduced in version 4.2.0. #### [](#parameters-111)Parameters | Name | Type | Description | | --- | --- | --- | | format (optional) | string | The output format using Go’s reference time. Default: "2006-01-02T15:04:05.999999999Z07:00". | | tz (optional) | string | Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. | #### [](#examples-154)Examples Format timestamp with custom format: ```bloblang root.something_at = this.created_at.ts_format("2006-Jan-02 15:04:05") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} ``` Format unix timestamp with timezone specification: ```bloblang root.something_at = this.created_at.ts_format(format: "2006-Jan-02 15:04:05", tz: "UTC") # In: {"created_at":1597405526} # Out: {"something_at":"2020-Aug-14 11:45:26"} ``` ### [](#ts_parse)ts_parse > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts\_strptime for strftime-style formats instead. Introduced in version 4.2.0. #### [](#parameters-112)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format of the input string using Go’s reference time. | #### [](#examples-155)Examples Parse a date with abbreviated month name: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_parse("2006-Jan-02") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} ``` Parse a custom datetime format: ```bloblang root.parsed = this.timestamp.ts_parse("Jan 2, 2006 at 3:04pm (MST)") # In: {"timestamp":"Aug 14, 2020 at 5:54am (UTC)"} # Out: {"parsed":"2020-08-14T05:54:00Z"} ``` ### [](#ts_round)ts_round > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Rounds a timestamp to the nearest multiple of the specified duration. Halfway values round up. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. Introduced in version 4.2.0. #### [](#parameters-113)Parameters | Name | Type | Description | | --- | --- | --- | | duration | integer | A duration measured in nanoseconds to round by. | #### [](#examples-156)Examples Round timestamp to the nearest hour: ```bloblang root.created_at_hour = this.created_at.ts_round("1h".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_hour":"2020-08-14T06:00:00Z"} ``` Round timestamp to the nearest minute: ```bloblang root.created_at_minute = this.created_at.ts_round("1m".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_minute":"2020-08-14T05:54:00Z"} ``` ### [](#ts_strftime)ts_strftime > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts\_format for Go-style reference time formats. Introduced in version 4.2.0. #### [](#parameters-114)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using strptime specifiers. | | tz (optional) | string | Optional timezone. Defaults to input timezone or local time for unix timestamps. | #### [](#examples-157)Examples Format timestamp with strftime specifiers: ```bloblang root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} ``` Format with microseconds using %f directive: ```bloblang root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S.%f", "UTC") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26.371000"} ``` ### [](#ts_strptime)ts_strptime > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts\_parse for Go-style reference time formats instead. Introduced in version 4.2.0. #### [](#parameters-115)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format string using strptime specifiers (e.g., %Y-%m-%d). | #### [](#examples-158)Examples Parse date with abbreviated month using strptime format: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} ``` Parse datetime with microseconds using %f directive: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d %H:%M:%S.%f") # In: {"doc":{"timestamp":"2020-Aug-14 11:50:26.371000"}} # Out: {"doc":{"timestamp":"2020-08-14T11:50:26.371Z"}} ``` ### [](#ts_sub)ts_sub > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Calculates the duration in nanoseconds between two timestamps (t1 - t2). Returns a signed integer: positive if t1 is after t2, negative if t1 is before t2. Use .abs() for absolute duration. Introduced in version 4.23.0. #### [](#parameters-116)Parameters | Name | Type | Description | | --- | --- | --- | | t2 | timestamp | The timestamp to subtract from the target timestamp. | #### [](#examples-159)Examples Calculate absolute duration between two timestamps: ```bloblang root.between = this.started_at.ts_sub("2020-08-14T05:54:23Z").abs() # In: {"started_at":"2020-08-13T05:54:23Z"} # Out: {"between":86400000000000} ``` Calculate signed duration (can be negative): ```bloblang root.duration_ns = this.end_time.ts_sub(this.start_time) # In: {"start_time":"2020-08-14T10:00:00Z","end_time":"2020-08-14T11:30:00Z"} # Out: {"duration_ns":5400000000000} ``` ### [](#ts_sub_iso8601)ts_sub_iso8601 > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Subtracts an ISO 8601 duration from a timestamp with calendar-aware precision for years, months, and days. Useful when you need to subtract durations that account for variable month lengths or leap years. Introduced in version 4.14.0. #### [](#parameters-117)Parameters | Name | Type | Description | | --- | --- | --- | | duration | string | Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) | #### [](#examples-160)Examples Subtract one year from a timestamp: ```bloblang root.last_year = this.created_at.ts_sub_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"last_year":"2019-08-14T05:54:23Z"} ``` Subtract a complex duration with multiple units: ```bloblang root.past_date = this.created_at.ts_sub_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2021-03-04T04:05:06Z"} # Out: {"past_date":"2020-01-01T00:00:00Z"} ``` ### [](#ts_tz)ts_tz > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Converts a timestamp to a different timezone while preserving the moment in time. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. Introduced in version 4.3.0. #### [](#parameters-118)Parameters | Name | Type | Description | | --- | --- | --- | | tz | string | The timezone to change to. Use "UTC" for UTC, "Local" for local timezone, or an IANA Time Zone database location name like "America/New_York". | #### [](#examples-161)Examples Convert timestamp to UTC timezone: ```bloblang root.created_at_utc = this.created_at.ts_tz("UTC") # In: {"created_at":"2021-02-03T17:05:06+01:00"} # Out: {"created_at_utc":"2021-02-03T16:05:06Z"} ``` Convert timestamp to a specific timezone: ```bloblang root.created_at_ny = this.created_at.ts_tz("America/New_York") # In: {"created_at":"2021-02-03T16:05:06Z"} # Out: {"created_at_ny":"2021-02-03T11:05:06-05:00"} ``` ### [](#ts_unix)ts_unix > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. Introduced in version 4.2.0. #### [](#examples-162)Examples Convert RFC 3339 timestamp to unix seconds: ```bloblang root.created_at_unix = this.created_at.ts_unix() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000} ``` Unix timestamp passthrough returns same value: ```bloblang root.timestamp = this.ts.ts_unix() # In: {"ts":1257894000} # Out: {"timestamp":1257894000} ``` ### [](#ts_unix_micro)ts_unix_micro > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. Introduced in version 4.12.0. #### [](#examples-163)Examples Convert timestamp to microseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_micro() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000} ``` Preserve microsecond precision from timestamp: ```bloblang root.precise_time = this.timestamp.ts_unix_micro() # In: {"timestamp":"2020-08-14T11:45:26.123456Z"} # Out: {"precise_time":1597405526123456} ``` ### [](#ts_unix_milli)ts_unix_milli > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. Introduced in version 4.12.0. #### [](#examples-164)Examples Convert timestamp to milliseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_milli() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000} ``` Useful for JavaScript timestamp compatibility: ```bloblang root.js_timestamp = this.event_time.ts_unix_milli() # In: {"event_time":"2020-08-14T11:45:26.123Z"} # Out: {"js_timestamp":1597405526123} ``` ### [](#ts_unix_nano)ts_unix_nano > ⚠️ **CAUTION** > > This method is in beta. Its behavior might change in a future release. Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. Introduced in version 4.2.0. #### [](#examples-165)Examples Convert timestamp to nanoseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_nano() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000000} ``` Preserve full nanosecond precision: ```bloblang root.precise_time = this.timestamp.ts_unix_nano() # In: {"timestamp":"2020-08-14T11:45:26.123456789Z"} # Out: {"precise_time":1597405526123456789} ``` ## [](#type-coercion)Type coercion ### [](#array)array Return an array containing the target value. If the value is already an array it is unchanged. #### [](#examples-166)Examples ```bloblang root.my_array = this.name.array() # In: {"name":"foobar bazson"} # Out: {"my_array":["foobar bazson"]} ``` ### [](#bool)bool Attempt to parse a value into a boolean. An optional argument can be provided, in which case if the value cannot be parsed the argument will be returned instead. If the value is a number then any non-zero value will resolve to `true`, if the value is a string then any of the following values are considered valid: `1, t, T, TRUE, true, True, 0, f, F, FALSE`. #### [](#parameters-119)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | bool | An optional value to yield if the target cannot be parsed as a boolean. | #### [](#examples-167)Examples ```bloblang root.foo = this.thing.bool() root.bar = this.thing.bool(true) ``` ### [](#bytes)bytes Marshal a value into a byte array. If the value is already a byte array it is unchanged. #### [](#examples-168)Examples ```bloblang root.first_byte = this.name.bytes().index(0) # In: {"name":"foobar bazson"} # Out: {"first_byte":102} ``` ### [](#not_empty)not_empty Ensures that the given string, array or object value is not empty, and if so returns it, otherwise an error is returned. #### [](#examples-169)Examples ```bloblang root.a = this.a.not_empty() # In: {"a":"foo"} # Out: {"a":"foo"} # In: {"a":""} # Out: Error("failed assignment (line 1): field `this.a`: string value is empty") # In: {"a":["foo","bar"]} # Out: {"a":["foo","bar"]} # In: {"a":[]} # Out: Error("failed assignment (line 1): field `this.a`: array value is empty") # In: {"a":{"b":"foo","c":"bar"}} # Out: {"a":{"b":"foo","c":"bar"}} # In: {"a":{}} # Out: Error("failed assignment (line 1): field `this.a`: object value is empty") ``` ### [](#not_null)not_null Ensures that the given value is not `null`, and if so returns it, otherwise an error is returned. #### [](#examples-170)Examples ```bloblang root.a = this.a.not_null() # In: {"a":"foobar","b":"barbaz"} # Out: {"a":"foobar"} # In: {"b":"barbaz"} # Out: Error("failed assignment (line 1): field `this.a`: value is null") ``` ### [](#number)number Attempt to parse a value into a number. An optional argument can be provided, in which case if the value cannot be parsed into a number the argument will be returned instead. #### [](#parameters-120)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | float | An optional value to yield if the target cannot be parsed as a number. | #### [](#examples-171)Examples ```bloblang root.foo = this.thing.number() + 10 root.bar = this.thing.number(5) * 10 ``` ### [](#string)string Converts any value to its string representation. Numbers, booleans, and objects are converted to strings; existing strings are unchanged. Use for type coercion or creating string representations. #### [](#examples-172)Examples ```bloblang root.nested_json = this.string() # In: {"foo":"bar"} # Out: {"nested_json":"{\"foo\":\"bar\"}"} ``` ```bloblang root.id = this.id.string() # In: {"id":228930314431312345} # Out: {"id":"228930314431312345"} ``` ### [](#timestamp)timestamp Attempt to parse a value into a timestamp. An optional argument can be provided, in which case if the value cannot be parsed into a timestamp the argument will be returned instead. #### [](#parameters-121)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | timestamp | An optional value to yield if the target cannot be parsed as a timestamp. | #### [](#examples-173)Examples ```bloblang root.foo = this.ts.timestamp() root.bar = this.none.timestamp(1234567890.timestamp()) ``` ### [](#type)type Returns the type of a value as a string, providing one of the following values: `string`, `bytes`, `number`, `bool`, `timestamp`, `array`, `object` or `null`. #### [](#examples-174)Examples ```bloblang root.bar_type = this.bar.type() root.foo_type = this.foo.type() # In: {"bar":10,"foo":"is a string"} # Out: {"bar_type":"number","foo_type":"string"} ``` ```bloblang root.type = this.type() # In: "foobar" # Out: {"type":"string"} # In: 666 # Out: {"type":"number"} # In: false # Out: {"type":"bool"} # In: ["foo", "bar"] # Out: {"type":"array"} # In: {"foo": "bar"} # Out: {"type":"object"} # In: null # Out: {"type":"null"} ``` ```bloblang root.type = content().type() # In: foobar # Out: {"type":"bytes"} ``` ```bloblang root.type = this.ts_parse("2006-01-02").type() # In: "2022-06-06" # Out: {"type":"timestamp"} ``` ## [](#deprecated)Deprecated ### [](#format_timestamp)format_timestamp > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts\_strftime for strftime-style formats. #### [](#parameters-122)Parameters | Name | Type | Description | | --- | --- | --- | | format (optional) | string | The output format using Go’s reference time. Default: "2006-01-02T15:04:05.999999999Z07:00". | | tz (optional) | string | Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. | ### [](#format_timestamp_strftime)format_timestamp_strftime > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts\_format for Go-style reference time formats. #### [](#parameters-123)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using strptime specifiers. | | tz (optional) | string | Optional timezone. Defaults to input timezone or local time for unix timestamps. | ### [](#format_timestamp_unix)format_timestamp_unix > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. ### [](#format_timestamp_unix_micro)format_timestamp_unix_micro > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. Introduced in version 4.12.0. ### [](#format_timestamp_unix_milli)format_timestamp_unix_milli > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. Introduced in version 4.12.0. ### [](#format_timestamp_unix_nano)format_timestamp_unix_nano > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. ### [](#parse_timestamp)parse_timestamp > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts\_strptime for strftime-style formats instead. #### [](#parameters-124)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format of the input string using Go’s reference time. | ### [](#parse_timestamp_strptime)parse_timestamp_strptime > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts\_parse for Go-style reference time formats instead. #### [](#parameters-125)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format string using strptime specifiers (e.g., %Y-%m-%d). | --- # Page 381: Bloblang Playground **URL**: https://docs.redpanda.com/connect/guides/bloblang/playground.md --- # Bloblang Playground --- title: Bloblang Playground latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/playground page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/playground.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/playground.adoc description: Experiment with Bloblang in this interactive playground. Test mappings, process JSON inputs, and explore examples of real-time data transformations. page-git-created-date: "2024-12-03" page-git-modified-date: "2024-12-03" --- Enter your data in **Input**, write a Bloblang mapping to transform it, and see the result in **Output**. Or, [learn Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/). × ### Keyboard Shortcuts × - Ctrl/Cmd + Enter Execute mapping - Ctrl/Cmd + Shift + X Clear all fields - Ctrl/Cmd + Shift + F Format JSON input - Ctrl/Cmd + S Copy share link - Ctrl/Cmd + E Export configuration - Ctrl/Cmd + / Toggle this help #### Claude AI Assistant Use Claude Code to help with Bloblang transformations - **/rpcn:blobl** - Generate mappings from natural language - **/rpcn:search** - Discover Redpanda Connect components - **/rpcn:pipeline** - Build complete streaming pipelines [Learn more about Claude Code for Redpanda →](https://github.com/redpanda-data/connect/blob/main/.claude-plugin/README.md) Access array items Add metadata Add new field Aggregation Array processing Conditional mapping Hello world Merge objects Nested object Remove sensitive data String manipulations Share Export Import ? Run Auto-run ⓘ #### Input ⓘ Auto-detect JSON Text CSV XML Ask AI Format Copy Clear #### Mapping ⓘ Ask AI Copy Clear ↓ #### Output ⓘ Copy Output will appear here... ▶ Advanced Options Optional #### Input Metadata Format Copy Clear #### Output Metadata Metadata will appear here... --- # Page 382: Bloblang Walkthrough **URL**: https://docs.redpanda.com/connect/guides/bloblang/walkthrough.md --- # Bloblang Walkthrough --- title: Bloblang Walkthrough latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: bloblang/walkthrough page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/walkthrough.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/walkthrough.adoc description: A step by step introduction to Bloblang page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-14" --- Bloblang is the most advanced mapping language that you’ll learn from this walkthrough (probably). It is designed for readability, the power to shape even the most outrageous input documents, and to easily make erratic schemas bend to your will. Bloblang is the native mapping language of Redpanda Connect, but it has been designed as a general purpose technology ready to be adopted by other tools. In this walkthrough you’ll learn how to make new friends by mapping their documents, and lose old friends as they grow jealous and bitter of your mapping abilities. There are a few ways to execute Bloblang but the way we’ll do it in this guide is to pull a Redpanda Connect docker image and run the command `rpk connect blobl server`, which opens up an interactive Bloblang editor: ```sh docker pull docker.redpanda.com/redpandadata/connect:latest docker run -p 4195:4195 --rm docker.redpanda.com/redpandadata/connect blobl server --no-open --host 0.0.0.0 ``` > 📝 **NOTE: Alternatives** > > Alternatives > > For alternative Redpanda Connect installation options check out the [getting started guide](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/). Next, open your browser at `http://localhost:4195` and you should see an app with three panels, the top-left is where you paste an input document, the bottom is your Bloblang mapping and on the top-right is the output. ## [](#your-first-assignment)Your first assignment The primary goal of a Bloblang mapping is to construct a brand new document by using an input document as a reference, which we achieve through a series of assignments. Bloblang is traditionally used to map JSON documents and that’s mostly what we’ll be doing in this walkthrough. The first mapping you’ll see when you open the editor is a single assignment: ```bloblang root = this # In: {"message":"hello world"} # Out: {"message":"hello world"} ``` On the left-hand side of the assignment is our assignment target, where `root` is a keyword referring to the root of the new document being constructed. On the right-hand side is a query which determines the value to be assigned, where `this` is a keyword that refers to the context of the mapping which begins as the root of the input document. As you can see the input document in the editor begins as a JSON object `{"message":"hello world"}`, and the output panel should show the result as: ```json { "message": "hello world" } ``` This output is a (neatly formatted) replica of the input document. This is the result of our mapping because we assigned the entire input document to the root of our new thing. Let’s create a brand new document by assigning a fresh object to the root: ```bloblang root = {} root.foo = this.message # In: {"message":"hello world"} # Out: {"foo":"hello world"} ``` Bloblang supports a bunch of [literal types](https://docs.redpanda.com/connect/guides/bloblang/about/#literals), and the first line of this mapping assigns an empty object literal to the root. The second line then creates a new field `foo` on that object by assigning it the value of `message` from the input document. You should see that our output has changed to: ```json { "foo": "hello world" } ``` In Bloblang, when the path that we assign to contains fields that are themselves unset then they are created as empty objects. This rule also applies to `root` itself, which means the mapping: ```bloblang root.foo.bar = this.message root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"hello world","buz me":{"baz":"I like mapping"}}} ``` Will automatically create the objects required to produce the output document: ```json { "foo": { "bar": "hello world", "buz me": { "baz": "I like mapping" } } } ``` Also note that we can use quotes in order to express path segments that contain symbols or whitespace. Great, let’s move on quick before our self-satisfaction gets in the way of progress. ## [](#basic-methods-and-functions)Basic methods and functions Nothing is ever good enough for you, why should the input document be any different? Usually in our mappings it’s necessary to mutate values whilst we map them over, this is almost always done with methods, of which [there are many](https://docs.redpanda.com/connect/guides/bloblang/methods/). To demonstrate we’re going to change our mapping to [uppercase](https://docs.redpanda.com/connect/guides/bloblang/methods/#uppercase) the field `message` from our input document: ```bloblang root.foo.bar = this.message.uppercase() root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO WORLD","buz me":{"baz":"I like mapping"}}} ``` As you can see the syntax for a method is similar to many languages, simply add a dot on the target value followed by the method name and arguments within brackets. With this method added our output document should look like this: ```json { "foo": { "bar": "HELLO WORLD", "buz me": { "baz": "I like mapping" } } } ``` Since the result of any Bloblang query is a value you can use methods on anything, including other methods. For example, we could expand our mapping of `message` to also replace `WORLD` with `EARTH` using the [`replace_all` method](https://docs.redpanda.com/connect/guides/bloblang/methods/#replace_all): ```bloblang root.foo.bar = this.message.uppercase().replace_all("WORLD", "EARTH") root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO EARTH","buz me":{"baz":"I like mapping"}}} ``` As you can see this method required some arguments. Methods support both nameless (like above) and named arguments, which are often literal values but can also be queries themselves. For example try out the following mapping using both named style and a dynamic argument: ```bloblang root.foo.bar = this.message.uppercase().replace_all(old: "WORLD", new: this.message.capitalize()) root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO Hello World","buz me":{"baz":"I like mapping"}}} ``` Woah, I think that’s the plot to Inception, let’s move onto functions. Functions are just boring methods that don’t have a target, and there are [plenty of them as well](https://docs.redpanda.com/connect/guides/bloblang/functions/). Functions are often used to extract information unrelated to the input document, such as [environment variables](https://docs.redpanda.com/connect/guides/bloblang/functions/#env), or to generate data such as [timestamps](https://docs.redpanda.com/connect/guides/bloblang/functions/#now) or [UUIDs](https://docs.redpanda.com/connect/guides/bloblang/functions/#uuid_v4). Since we’re completionists let’s add one to our mapping: ```bloblang root.foo.bar = this.message.uppercase().replace_all("WORLD", "EARTH") root.foo."buz me".baz = "I like mapping" root.foo.id = uuid_v4() # In: {"message":"hello world"} ``` Now I can’t tell you what the output looks like since it will be different each time it’s mapped, how fun! ### [](#deletions)Deletions Everything in Bloblang is an expression to be assigned, including deletions, which is a [function `deleted()`](https://docs.redpanda.com/connect/guides/bloblang/functions/#deleted). To illustrate let’s create a field we want to delete by changing our input to the following: ```json { "name": "fooman barson", "age": 7, "opinions": ["trucks are cool","trains are cool","chores are bad"] } ``` If we wanted a full copy of this document without the field `name` then we can assign `deleted()` to it: ```bloblang root = this root.name = deleted() # In: {"name":"fooman barson","age":7,"opinions":["trucks are cool","trains are cool","chores are bad"]} # Out: {"age":7,"opinions":["trucks are cool","trains are cool","chores are bad"]} ``` And it won’t be included in the output: ```json { "age": 7, "opinions": [ "trucks are cool", "trains are cool", "chores are bad" ] } ``` An alternative way to delete fields is the [method `without`](https://docs.redpanda.com/connect/guides/bloblang/methods/#without), our above example could be rewritten as a single assignment `root = this.without("name")`. However, `deleted()` is generally more powerful and will come into play more later on. ## [](#variables)Variables Sometimes it’s necessary to capture a value for later, but we might not want it to be added to the resulting document. In Bloblang we can achieve this with variables which are created using the `let` keyword, and can be referenced within subsequent queries with a dollar sign prefix: ```bloblang let id = uuid_v4() root.id_sha1 = $id.hash("sha1").encode("hex") root.id_md5 = $id.hash("md5").encode("hex") # In: {} ``` Variables can be assigned any value type, including objects and arrays. ## [](#unstructured-and-binary-data)Unstructured and binary data So far in all of our examples both the input document and our newly mapped document are structured, but this does not need to be so. Try assigning some literal value types directly to the `root`, such as a string `root = "hello world"`, or a number `root = 5`. You should notice that when a value type is assigned to the root the output is the raw value, and therefore strings are not quoted. This is what makes it possible to output data of any format, including encrypted, encoded or otherwise binary data. Unstructured mapping is not limited to the output. Rather than referencing the input document with `this`, where it must be structured, it is possible to reference it as a binary string with the [function `content`](https://docs.redpanda.com/connect/guides/bloblang/functions/#content), try changing your mapping to: ```bloblang root = content().uppercase() # In: hello world # Out: HELLO WORLD ``` When you add content to the input panel, it should be the same in the output panel, but in all uppercase. ## [](#conditionals)Conditionals In order to play around with conditionals let’s set our input to something structured: ```json { "pet": { "type": "cat", "is_cute": true, "treats": 5, "toys": 3 } } ``` In Bloblang all conditionals are expressions, this is a core principal of Bloblang and will be important later on when we’re mapping deeply nested structures. ### [](#if-expression)If expression The simplest conditional is the `if` expression, where the boolean condition does not need to be in parentheses. Let’s create a map that modifies the number of treats our pet receives based on a field: ```bloblang root = this root.pet.treats = if this.pet.is_cute { this.pet.treats + 10 } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` Try that mapping out and you should see the number of treats in the output increased to 15. Now try changing the input field `pet.is_cute` to `false` and the output treats count should go back to the original 5. When a conditional expression doesn’t have a branch to execute then the assignment is skipped entirely, which means when the pet is not cute the value of `pet.treats` is unchanged (and remains the value set in the `root = this` assignment). We can add an `else` block to our `if` expression to remove treats entirely when the pet is not cute: ```bloblang root = this root.pet.treats = if this.pet.is_cute { this.pet.treats + 10 } else { deleted() } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` This is possible because field deletions are expressed as assigned values created with the `deleted()` function. ### [](#if-statement)If statement The `if` keyword can also be used as a statement in order to conditionally apply a series of mapping assignments, the previous example can be rewritten as: ```bloblang root = this if this.pet.is_cute { root.pet.treats = this.pet.treats + 10 } else { root.pet.treats = deleted() } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` Converting this mapping to use a statement has resulted in a more verbose mapping as we had to specify `root.pet.treats` multiple times as an assignment target. However, using `if` as a statement can be beneficial when multiple assignments rely on the same logic: ```bloblang root = this if this.pet.is_cute { root.pet.treats = this.pet.treats + 10 root.pet.toys = this.pet.toys + 10 } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":13}} ``` More treats _and_ more toys! Lucky Spot! ### [](#match-expression)Match expression Another conditional expression is `match` which allows you to list many branches consisting of a condition and a query to execute separated with `=>`, where the first condition to pass is the one that is executed: ```bloblang root = this root.pet.toys = match { this.pet.treats > 5 => this.pet.treats - 5, this.pet.type == "cat" => 3, this.pet.type == "dog" => this.pet.toys - 3, this.pet.type == "horse" => this.pet.toys + 10, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` Try executing that mapping with different values for `pet.type` and `pet.treats`. Match expressions can also specify a new context for the keyword `this` which can help reduce some of the boilerplate in your boolean conditions. The following mapping is equivalent to the previous: ```bloblang root = this root.pet.toys = match this.pet { this.treats > 5 => this.treats - 5, this.type == "cat" => 3, this.type == "dog" => this.toys - 3, this.type == "horse" => this.toys + 10, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` Your boolean conditions can also be expressed as value types, in which case the context being matched will be compared to the value: ```bloblang root = this root.pet.toys = match this.pet.type { "cat" => 3, "dog" => 5, "rabbit" => 8, "horse" => 20, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` ## [](#error-handling)Error handling Bloblang can simplify handling errors. First, let’s take a look at what happens when errors _aren’t_ handled, change your input to the following: ```json { "palace_guards": 10, "angry_peasants": "I couldn't be bothered to ask them" } ``` And change your mapping to something simple like a number comparison: ```bloblang root.in_trouble = this.angry_peasants > this.palace_guards # In: {"palace_guards":10,"angry_peasants":"I couldn't be bothered to ask them"} ``` Uh oh! It looks like our canvasser was too lazy and our `angry_peasants` count was incorrectly set for this document. You should see an error in the output window that mentions something like `cannot compare types string (from field this.angry_peasants) and number (from field this.palace_guards)`, which means the mapping was abandoned. So what if we want to try and map something, but don’t care if it fails? In this case if we are unable to compare our angry peasants with palace guards then I would still consider us in trouble just to be safe. For that we have a special [method `catch`](https://docs.redpanda.com/connect/guides/bloblang/methods/#catch), which if we add to any query allows us to specify an argument to be returned when an error occurs. Since methods can be added to any query we can surround our arithmetic with brackets and catch the whole thing: ```bloblang root.in_trouble = (this.angry_peasants > this.palace_guards).catch(true) # In: {"palace_guards":10,"angry_peasants":"I couldn't be bothered to ask them"} # Out: {"in_trouble":true} ``` Now instead of an error we should see an output with `in_trouble` set to `true`. Try changing to value of `angry_peasants` to a few different values, including some numbers. One of the powerful features of `catch` is that when it is added at the end of a series of expressions and methods it will capture errors at any part of the series, allowing you to capture errors at any granularity. For example, the mapping: ```bloblang root.abort_mission = if this.mission.type == "impossible" { !this.user.motives.contains("must clear name") } else { this.mission.difficulty > 10 }.catch(false) # In: {"mission":{"type":"impossible","difficulty":5},"user":{"motives":["must clear name"]}} # Out: {"abort_mission":false} ``` Will catch errors caused by: - `this.mission.type` not being a string - `this.user.motives` not being an array - `this.mission.difficulty` not being a number But will always return `false` if any of those errors occur. Try it out with this input and play around by breaking some of the fields: ```json { "mission": { "type": "impossible", "difficulty": 5 }, "user": { "motives": ["must clear name"] } } ``` Now try out this mapping: ```bloblang root.abort_mission = if (this.mission.type == "impossible").catch(true) { !this.user.motives.contains("must clear name").catch(false) } else { (this.mission.difficulty > 10).catch(true) } # In: {"mission":{"type":"impossible","difficulty":5},"user":{"motives":["must clear name"]}} # Out: {"abort_mission":false} ``` This version is more granular and will capture each of the errors individually, with each error given a unique `true` or `false` fallback. ## [](#validation)Validation Sometimes errors are what we want. Failing a mapping with an error allows us to handle the bad document in other ways, such as routing it to a dead-letter queue or filtering it entirely. You can read about common Redpanda Connect error handling patterns for bad data in the [error handling guide](https://docs.redpanda.com/connect/configuration/error_handling/), but the first step is to create the error. Luckily, Bloblang has a range of ways of creating errors under certain circumstances, which can be used in order to validate the data being mapped. There are [a few helper methods](https://docs.redpanda.com/connect/guides/bloblang/methods/#type-coercion) that make validating and coercing fields nice and easy, try this mapping out: ```bloblang root.foo = this.foo.number() root.bar = this.bar.not_null() root.baz = this.baz.not_empty() # In: {"foo":5,"bar":"hello world","baz":[1,2,3]} # Out: {"foo":5,"bar":"hello world","baz":[1,2,3]} ``` With some of these sample inputs: ```json {"foo":"nope","bar":"hello world","baz":[1,2,3]} {"foo":5,"baz":[1,2,3]} {"foo":10,"bar":"hello world","baz":[]} ``` However, these methods don’t cover all use cases. The general purpose error throwing technique is the [`throw` function](https://docs.redpanda.com/connect/guides/bloblang/functions/#throw), which takes an argument string that describes the error. When it’s called it will throw a mapping error that abandons the mapping. For example, we can check the type of a field with the [method `type`](https://docs.redpanda.com/connect/guides/bloblang/methods/#type), and then throw an error if it’s not the type we expected: ```bloblang root.foos = if this.user.foos.type() == "array" { this.user.foos } else { throw("foos must be an array, but it ain't, what gives?") } # In: {"user":{"foos":[1,2,3]}} ``` Try this mapping out with a few sample inputs: ```json {"user":{"foos":[1,2,3]}} {"user":{"foos":"1,2,3"}} ``` ## [](#context)Context In Bloblang, when we refer to the context we’re talking about the value returned with the keyword `this`. At the beginning of a mapping the context starts off as a reference to the root of a structured input document, which is why the mapping `root = this` will result in the same document coming out as you put in. However, in Bloblang there are mechanisms whereby the context might change, we’ve already seen how this can happen within a `match` expression. Another useful way to change the context is by adding a bracketed query expression as a method to a query, which looks like this: ```bloblang root = this.foo.bar.(this.baz + this.buz) # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` Within the bracketed query expression the context becomes the result of the query that it’s a method of, so within the brackets in the above mapping the value of `this` points to the result of `this.foo.bar`, and the mapping is therefore equivalent to: ```bloblang root = this.foo.bar.baz + this.foo.bar.buz # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` With this handy trick the `throw` mapping from the validation section above could be rewritten as: ```bloblang root.foos = this.user.foos.(if this.type() == "array" { this } else { throw("foos must be an array, but it ain't, what gives?") }) # In: {"user":{"foos":[1,2,3]}} # Out: {"foos":[1,2,3]} ``` ### [](#naming-the-context)Naming the context Shadowing the keyword `this` with new contexts can look confusing in your mappings, and it also limits you to only being able to reference one context at any given time. As an alternative, Bloblang supports context capture expressions that look similar to lambda functions from other languages, where you can name the new context with the syntax ` -> `, which looks like this: ```bloblang root = this.foo.bar.(thing -> thing.baz + thing.buz) # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` Within the brackets we now have a new field `thing`, which returns the context that would have otherwise been captured as `this`. This also means the value returned from `this` hasn’t changed and will continue to return the root of the input document. ## [](#coalescing)Coalescing Being able to open up bracketed query expressions on fields leads us onto another cool trick in Bloblang referred to as coalescing. It’s very common in the world of document mapping that due to structural deviations a value that we wish to obtain could come from one of multiple possible paths. To illustrate this problem change the input document to the following: ```json { "thing": { "article": { "id": "foo", "contents": "Some people did some stuff" } } } ``` Let’s say we wish to flatten this structure with the following mapping: ```bloblang root.contents = this.thing.article.contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` But articles are only one of many document types we expect to receive, where the field `contents` remains the same but the field `article` could instead be `comment` or `share`. In this case we could expand our map of `contents` to use a `match` expression where we check for the existence of `article`, `comment`, etc in the input document. However, a much cleaner way of approaching this is with the pipe operator (`|`), which in Bloblang can be used to join multiple queries, where the first to yield a non-null result is selected. Change your mapping to the following: ```bloblang root.contents = this.thing.article.contents | this.thing.comment.contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` And now try changing the field `article` in your input document to `comment`. You should see that the value of `contents` remains as `Some people did some stuff` in the output document. Now, rather than write out the full path prefix `this.thing` each time we can use a bracketed query expression to change the context, giving us more space for adding other fields: ```bloblang root.contents = this.thing.(this.article | this.comment | this.share).contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` And by the way, the keyword `this` within queries can be omitted and made implicit, which allows us to reduce this even further: ```bloblang root.contents = this.thing.(article | comment | share).contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` Finally, we can also add a pipe operator at the end to fallback to a literal value when none of our candidates exists: ```bloblang root.contents = this.thing.(article | comment | share).contents | "nothing" # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` Neat. ## [](#advanced-methods)Advanced methods What happens when you need to map all of the elements of an array? Or filter the keys of an object by their values? What if the fellowship just used the eagles to fly to mount doom? Bloblang offers a bunch of advanced methods for [manipulating structured data types](https://docs.redpanda.com/connect/guides/bloblang/methods/#object-array-manipulation), let’s take a quick tour of some of the cooler ones. Set your input document to this list of things: ```json { "num_friends": 5, "things": [ { "name": "yo-yo", "quantity": 10, "is_cool": true }, { "name": "dish soap", "quantity": 50, "is_cool": false }, { "name": "scooter", "quantity": 1, "is_cool": true }, { "name": "pirate hat", "quantity": 7, "is_cool": true } ] } ``` Let’s say we wanted to reduce the `things` in our input document to only those that are cool and where we have enough of them to share with our friends. We can do this with a [`filter` method](https://docs.redpanda.com/connect/guides/bloblang/methods/#filter): ```bloblang root = this.things.filter(thing -> thing.is_cool && thing.quantity > this.num_friends) # In: {"num_friends":5,"things":[{"name":"yo-yo","quantity":10,"is_cool":true},{"name":"dish soap","quantity":50,"is_cool":false},{"name":"scooter","quantity":1,"is_cool":true},{"name":"pirate hat","quantity":7,"is_cool":true}]} # Out: [{"name":"yo-yo","quantity":10,"is_cool":true},{"name":"pirate hat","quantity":7,"is_cool":true}] ``` Try running that mapping and you’ll see that the output is reduced. What is happening here is that the `filter` method takes an argument that is a query, and that query will be mapped for each individual element of the array (where the context is changed to the element itself). We have captured the context into a field `thing` which allows us to continue referencing the root of the input with `this`. The `filter` method requires the query parameter to resolve to a boolean `true` or `false`, and if it resolves to `true` the element will be present in the resulting array, otherwise it is removed. Being able to express a query argument to be applied to a range in this way is one of the more powerful features of Bloblang, and when mapping complex structured data these advanced methods will likely be a common tool that you’ll reach for. Another such method is [`map_each`](https://docs.redpanda.com/connect/guides/bloblang/methods/#map_each), which allows you to mutate each element of an array, or each value of an object. Change your input document to the following: ```json { "talking_heads": [ "1:E.T. is a bad film,Pokemon corrupted an entire generation", "2:Digimon ripped off Pokemon,Cats are boring", "3:I'm important", "4:Science is just made up,The Pokemon films are good,The weather is good" ] } ``` Here we have an array of talking heads, where each element is a string containing an identifer, a colon, and a comma separated list of their opinions. We wish to map each string into a structured object, which we can do with the following mapping: ```bloblang root = this.talking_heads.map_each(raw -> { "id": raw.split(":").index(0), "opinions": raw.split(":").index(1).split(",") }) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` The argument to `map_each` is a query where the context is the element, which we capture into the field `raw`. The result of the query argument will become the value of the element in the resulting array, and in this case we return an object literal. In order to separate the identifier from opinions we perform a `split` by colon on the raw string element and get the first substring with the `index` method. We then do the split again and extract the remainder, and split that by comma in order to extract all of the opinions to an array field. However, one problem with this mapping is that the split by colon is written out twice and executed twice. A more efficient way of performing the same thing is with the bracketed query expressions we’ve played with before: ```bloblang root = this.talking_heads.map_each(raw -> raw.split(":").(split_string -> { "id": split_string.index(0), "opinions": split_string.index(1).split(",") })) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` > 📝 **NOTE: Challenge!** > > Challenge! > > Try updating that map so that only opinions that mention Pokemon are kept To find more methods for manipulating structured data types check out the [methods page](https://docs.redpanda.com/connect/guides/bloblang/methods/#object-array-manipulation). ## [](#reusable-mappings)Reusable mappings Bloblang has cool methods, sure, but there’s nothing cooler than methods you’ve made yourself. When the going gets tough in the mapping world the best solution is often to create a named mapping, which you can do with the keyword `map`: ```bloblang map parse_talking_head { let split_string = this.split(":") root.id = $split_string.index(0) root.opinions = $split_string.index(1).split(",") } root = this.talking_heads.map_each(raw -> raw.apply("parse_talking_head")) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` The body of a named map, encapsulated with squiggly brackets, is a totally isolated mapping where `root` now refers to a new value being created for each invocation of the map, and `this` refers to the root of the context provided to the map. Named maps are executed with the [method `apply`](https://docs.redpanda.com/connect/guides/bloblang/methods/#apply), which has a string parameter identifying the map to execute, this means it’s possible to dynamically select the target map. As you can see in the above example we were able to use a custom map in order to create our talking head objects without the object literal. Within a named map we can also create variables that exist only within the scope of the map. A nice feature of named mappings is that they can invoke themselves recursively, allowing you to define mappings that walk deeply nested structures. The following mapping will scrub all values from a document that contain the word "Voldemort" (case insensitive): ```bloblang map remove_naughty_man { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, _ => this, } } root = this.apply("remove_naughty_man") # In: {"summer_party":{"theme":"the woman in black","guests":["Emma Bunton","the seal I spotted in Trebarwith","Voldemort","The cast of Swiss Army Man","Richard"],"notes":{"lisa":"I don't think voldemort eats fish","monty":"Seals hate dance music"}},"crushes":["Richard is nice but he hates pokemon","Victoria Beckham but I think she's taken","Charlie but they're totally into Voldemort"]} ``` Try running that mapping with the following input document: ```json { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith", "Voldemort", "The cast of Swiss Army Man", "Richard" ], "notes": { "lisa": "I don't think voldemort eats fish", "monty": "Seals hate dance music" } }, "crushes": [ "Richard is nice but he hates pokemon", "Victoria Beckham but I think she's taken", "Charlie but they're totally into Voldemort" ] } ``` ## [](#unit-testing)Unit testing Redpanda Connect has it’s own [unit testing capabilities](https://docs.redpanda.com/connect/configuration/unit_testing/) that you can also use for your mappings. To start with save a mapping into a file called something like `naughty_man.blobl`, we can use the example above from the reusable mappings section: ```bloblang map remove_naughty_man { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, _ => this, } } root = this.apply("remove_naughty_man") ``` Next, we can define our unit tests in an accompanying YAML file in the same directory, let’s call this `naughty_man_test.yaml`: ```yaml tests: - name: test naughty man scrubber target_mapping: './naughty_man.blobl' environment: {} input_batch: - content: | { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith", "Voldemort", "The cast of Swiss Army Man", "Richard" ] } } output_batches: - - json_equals: { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the dolphin I spotted in Trebarwith", "The cast of Swiss Army Man", "Richard" ] } } ``` As you can see we’ve defined a single test, where we point to our mapping file which will be executed in our test. We then specify an input message which is a reduced version of the document we tried out before, and finally we specify output predicates, which is a JSON comparison against the output document. We can execute these tests with `rpk connect test ./naughty_man_test.yaml`, Redpanda Connect will also automatically find our tests if you simply run `rpk connect test ./…​`. You should see an output something like: ```text Test 'naughty_man_test.yaml' failed Failures: --- naughty_man_test.yaml --- test naughty man scrubber [line 2]: batch 0 message 0: json_equals: JSON content mismatch { "summer_party": { "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith" => "the dolphin I spotted in Trebarwith", "The cast of Swiss Army Man", "Richard" ], "theme": "the woman in black" } } ``` Because in actual fact our expected output is wrong, I’ll leave it to you to spot the error. Once the test is fixed you should see: ```text Test 'naughty_man_test.yaml' succeeded ``` And now our mapping, should we need to expand it in the future, is better protected against regressions. You can read more about the Redpanda Connect unit test specification, including alternative output predicates, in [this document](https://docs.redpanda.com/connect/configuration/unit_testing/). --- # Page 383: Change Data Capture (CDC) **URL**: https://docs.redpanda.com/connect/guides/cdc.md --- # Change Data Capture (CDC) --- title: Change Data Capture (CDC) latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: cdc page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cdc.adoc description: Capture database changes in real time and stream them to Redpanda topics using native CDC inputs. page-topic-type: concept learning-objective-1: Describe how CDC inputs capture database changes in real time learning-objective-2: Compare Redpanda Connect CDC with Kafka Connect approaches learning-objective-3: Identify which CDC input to use for your database page-git-created-date: "2026-07-21" page-git-modified-date: "2026-10-14" --- Redpanda Connect captures row-level changes from databases in real time and streams them to Redpanda [topics](#topic). CDC inputs connect directly to database transaction logs or change streams, transforming each change into a [message](#message) that downstream applications can process immediately. After reading this page, you will be able to: - Describe how CDC inputs capture database changes in real time - Compare Redpanda Connect CDC with Kafka Connect approaches - Identify which CDC input to use for your database ## [](#common-use-cases)Common use cases CDC enables real-time data integration patterns: - Sync data between databases and data warehouses continuously, eliminating batch ETL delays - Build event-driven microservices that react to data changes within milliseconds - Replicate data across regions automatically for disaster recovery - Feed real-time analytics pipelines from operational databases without impacting performance - Keep search indexes synchronized with primary data stores without polling ## [](#cdc-inputs-in-redpanda-connect)CDC inputs in Redpanda Connect Redpanda Connect CDC inputs use declarative YAML configuration and integrate with [Bloblang transformations](https://docs.redpanda.com/connect/guides/bloblang/about/) for data manipulation. This approach eliminates the need for a separate Kafka Connect cluster, and handles schemas through built-in [schema registry processors](https://docs.redpanda.com/connect/components/processors/schema_registry_encode/). | Feature | Redpanda Connect CDC | Kafka Connect | | --- | --- | --- | | Configuration | Declarative YAML | JSON + REST API | | Transformations | Native Bloblang | SMTs (Single Message Transforms) | | Schema evolution | Built-in Schema Registry support | Requires additional configuration | | Deployment | Single binary, managed in Redpanda Cloud | Separate cluster required | For managed Redpanda Cloud deployments, CDC [pipelines](#pipeline) run as fully managed services. For self-hosted deployments, configure CDC inputs using [Redpanda Connect configuration files](https://docs.redpanda.com/connect/configuration/about/). ## [](#supported-databases)Supported databases Redpanda Connect provides CDC inputs for relational databases, NoSQL databases, and cloud services. ### [](#relational-databases)Relational databases - [PostgreSQL CDC](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) - Logical replication using replication slots - [MySQL CDC](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/) - Binary log CDC with GTID support - [Oracle CDC](https://docs.redpanda.com/connect/components/inputs/oracledb_cdc/) - LogMiner-based CDC for Oracle databases - [SQL Server CDC](https://docs.redpanda.com/connect/components/inputs/microsoft_sql_server_cdc/) - SQL Server change tracking ### [](#nosql-databases)NoSQL databases - [MongoDB CDC](https://docs.redpanda.com/connect/components/inputs/mongodb_cdc/) - Change streams for replica sets and sharded clusters - [DynamoDB CDC](https://docs.redpanda.com/connect/components/inputs/aws_dynamodb_cdc/) - DynamoDB Streams for real-time table changes ### [](#cloud-services)Cloud services - [Google Cloud Spanner CDC](https://docs.redpanda.com/connect/components/inputs/gcp_spanner_cdc/) - Spanner change streams - [Salesforce CDC](https://docs.redpanda.com/connect/components/inputs/salesforce_cdc/) - Pub/Sub API for Salesforce object changes - [TigerBeetle CDC](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/) - TigerBeetle accounting database change streams ## [](#example-patterns-and-configurations)Example patterns and configurations The following cookbooks demonstrate real-world CDC patterns with complete pipeline configurations. ### [](#oracle-cdc-pipeline)Oracle CDC pipeline See [Oracle CDC patterns](https://docs.redpanda.com/connect/cookbooks/oracledb_cdc/) for complete examples of filtering, transforming, and routing Oracle database changes. ### [](#dynamodb-cdc-pipeline)DynamoDB CDC pipeline See [DynamoDB CDC patterns](https://docs.redpanda.com/connect/cookbooks/dynamodb_cdc/) for patterns that stream DynamoDB table changes to Redpanda. ### [](#schema-registry-integration)Schema registry integration The PostgreSQL, MySQL, SQL Server, and Oracle CDC inputs attach each table’s schema to every message. See [CDC schema registry patterns](https://docs.redpanda.com/connect/cookbooks/cdc_schema_registry/) to register those schemas in a schema registry automatically and stream Avro-encoded change events to Redpanda. ## [](#suggested-reading)Suggested reading - [Transform CDC messages with Bloblang](https://docs.redpanda.com/connect/guides/bloblang/about/) - [Configure the Redpanda output](https://docs.redpanda.com/connect/components/outputs/redpanda/) - [Handle CDC pipeline failures](https://docs.redpanda.com/connect/configuration/error_handling/) - [Understand message delivery guarantees](https://docs.redpanda.com/connect/guides/delivery_semantics/) --- # Page 384: Cloud Credentials **URL**: https://docs.redpanda.com/connect/guides/cloud.md --- # Cloud Credentials --- title: Cloud Credentials latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: cloud/index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/index.adoc description: Explore how to deploy and manage Redpanda Connect in the cloud, including setup, configuration, and optimization tips. page-git-created-date: "2024-06-03" page-git-modified-date: "2024-06-04" --- - [Amazon Web Services](aws/) Find out about AWS components in Redpanda Connect. - [Authenticate to Amazon Aurora using IAM roles](aws-iam-aurora/) Configure IAM roles to authenticate self-hosted Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials. - [Google Cloud Platform](gcp/) Find out about GCP components in Redpanda Connect. --- # Page 385: Authenticate to Amazon Aurora using IAM roles **URL**: https://docs.redpanda.com/connect/guides/cloud/aws-iam-aurora.md --- # Authenticate to Amazon Aurora using IAM roles --- title: Authenticate to Amazon Aurora using IAM roles latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: cloud/aws-iam-aurora page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/aws-iam-aurora.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/aws-iam-aurora.adoc description: Configure IAM roles to authenticate self-hosted Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials. page-topic-type: how-to personas: streaming_developer, platform_operator learning-objective-1: Create an IAM database connect role with the required permission policy and trust relationship learning-objective-2: Grant your Redpanda Connect role cross-account access to Aurora learning-objective-3: Configure a Redpanda Connect pipeline input to authenticate to Aurora using IAM roles page-git-created-date: "2026-03-17" page-git-modified-date: "2026-08-07" --- Use IAM roles to authenticate self-hosted Redpanda Connect pipelines to Amazon Aurora on AWS without storing static credentials. > ❗ **IMPORTANT: Self-hosted deployments only** > > Self-hosted deployments only > > This guide applies to self-hosted Redpanda Connect deployments. Redpanda Cloud pipelines in Bring Your Own Cloud (BYOC) and Dedicated clusters do not currently support IAM Roles for Service Accounts (IRSA), and cross-account `sts:AssumeRole` from the Redpanda-managed pipeline role is not available because you cannot grant additional permissions to that role. The Redpanda Connect binary supports both, but the Cloud platform does not currently make pod identity tokens available to pipelines. To authenticate Cloud pipelines to AWS services, store static credentials as secrets and reference them in your pipeline configuration. See [Manage Secrets in Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/). Authentication uses a two-hop role chain. The Redpanda Connect process assumes an IAM role through its runtime environment, such as IRSA on Amazon EKS or an EC2 instance profile. That role then assumes a database-specific role in your Aurora account, which generates a short-lived RDS IAM token that Aurora accepts as a password. This page covers configuring the trust relationship and permissions between two roles that you own: the Connect role that your deployment runs as, and the database connect role in the Aurora account. The Connect role and its runtime binding (IRSA or an instance profile) must already exist. After reading this page, you will be able to: - Create an IAM database connect role with the required permission policy and trust relationship - Grant your Redpanda Connect role cross-account access to Aurora - Configure a Redpanda Connect pipeline input to authenticate to Aurora using IAM roles ## [](#prerequisites)Prerequisites - Redpanda Connect deployed on AWS infrastructure that you manage, such as Amazon EKS or EC2 - An Aurora cluster (PostgreSQL or MySQL) - Permissions to create IAM roles and attach inline policies in both the Connect and Aurora AWS accounts - An IAM role associated with the Redpanda Connect pod or EC2 instance (see [Identify your Redpanda Connect role](#identify-your-connect-role)) ## [](#identify-your-connect-role)Identify your Redpanda Connect role The Connect role is the IAM role that your Redpanda Connect process runs as: - On Amazon EKS, this is the IRSA role associated with the Kubernetes service account that runs your Redpanda Connect pods. To set up IRSA, see the [Amazon EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). - On EC2, this is the role attached to the instance profile. To set up an instance profile, see the [AWS IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html). The AWS SDK resolves this identity automatically through the default credential chain, so no credentials appear in your pipeline configuration. ## [](#find-aurora-cluster-resource-id)Find the Aurora cluster resource ID The policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Run the following command to retrieve the resource ID: ```bash aws rds describe-db-clusters \ --query "DBClusters[?DBClusterIdentifier==''].[DbClusterResourceId]" \ --output text \ --profile ``` The resource ID has the format `cluster-`. ## [](#create-database-connect-role)Create the database connect role In the AWS account where Aurora is hosted, create an IAM role with the following permission policy: permission-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Action": "rds-db:connect", "Effect": "Allow", "Resource": "arn:aws:rds-db:::dbuser:cluster-/" } ] } ``` Replace the following placeholders: - ``: The AWS region of the Aurora cluster - ``: The AWS account ID that hosts Aurora - ``: The cluster resource ID from [Find the Aurora cluster resource ID](#find-aurora-cluster-resource-id) - ``: The database user configured for IAM authentication ### [](#trust-policy)Trust policy Attach the following trust policy to the role to allow your Connect role to assume it: trust-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/" }, "Action": "sts:AssumeRole" } ] } ``` Replace the following placeholders: - ``: The AWS account ID where your Connect role lives - ``: The Connect role name from [Identify your Redpanda Connect role](#identify-your-connect-role) > 📝 **NOTE** > > The database connect role must be owned by the same AWS account as the Aurora instance. For cross-account setups, create this role in the Aurora account, not the account where Redpanda Connect runs. ### [](#create-the-role)Create the role Run the following AWS CLI commands to create the role and apply the permission policy: ```bash aws iam create-role \ --role-name "" \ --assume-role-policy-document file://trust-policy.json \ --profile aws iam put-role-policy \ --role-name "" \ --policy-name "aurora-rds-connect" \ --policy-document file://permission-policy.json \ --profile ``` ## [](#grant-connect-role-cross-account-access)Grant the Connect role cross-account access In the AWS account where your Connect role lives, add an inline policy to the Connect role (identified in [Identify your Redpanda Connect role](#identify-your-connect-role)) to allow it to assume your database connect role: inline-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam:::role/" } ] } ``` To attach the policy, run: ```bash aws iam put-role-policy \ --role-name "" \ --policy-name "allow-x-account-db-connect" \ --policy-document file://inline-policy.json \ --profile ``` To verify the policy is attached, run: ```bash aws iam get-role-policy \ --role-name "" \ --policy-name "allow-x-account-db-connect" \ --profile ``` ## [](#configure-the-network)Configure the network The Aurora security group must allow inbound traffic on the database port (5432 for PostgreSQL, 3306 for MySQL) from the source that Aurora sees: - For EKS or EC2 deployments in the same VPC as Aurora, allow the node or instance security group, or the private CIDR range that your Redpanda Connect pods or instances use. - For deployments that reach Aurora across a VPC boundary, such as through VPC peering or from another network, allow the IP addresses that your traffic originates from after translation, such as your NAT gateway IPs. ## [](#configure-the-pipeline)Configure the pipeline In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in [Create the database connect role](#create-database-connect-role). This configuration applies to the `postgres_cdc`, `pg_stream`, and `mysql_cdc` inputs. For the `postgres_cdc` and `pg_stream` inputs: ```yaml input: postgres_cdc: dsn: "host= port=5432 user= dbname= sslmode=require" aws: enabled: true region: endpoint: roles: - role: arn:aws:iam:::role/ ``` For the `mysql_cdc` input: ```yaml input: mysql_cdc: dsn: "@tcp(:3306)/?tls=true" aws: enabled: true region: endpoint: roles: - role: arn:aws:iam:::role/ ``` The `aws.roles` field accepts an array of role ARNs, which are assumed in sequence. This supports chaining multiple role assumptions for more complex cross-account setups. ## [](#troubleshooting)Troubleshooting These errors appear in the Redpanda Connect process logs. ### [](#err-sts-assumerole)`is not authorized to perform: sts:AssumeRole` The inline assume-role policy is missing from your Connect role, or the trust policy on the database connect role does not name your Connect role. 1. Verify that the database connect role’s trust policy lists your Connect role as a principal, as described in [Trust policy](#trust-policy). 2. Attach the inline policy as described in [Grant the Connect role cross-account access](#grant-connect-role-cross-account-access). The error clears automatically after the policy is in place. Look for a message like `postgres_cdc input go active` or `mysql_cdc input go active` in the logs to confirm successful authentication. ## [](#next-steps)Next steps - [`postgres_cdc` input](https://docs.redpanda.com/connect/components/inputs/postgres_cdc/) - [`mysql_cdc` input](https://docs.redpanda.com/connect/components/inputs/mysql_cdc/) --- # Page 386: Amazon Web Services **URL**: https://docs.redpanda.com/connect/guides/cloud/aws.md --- # Amazon Web Services --- title: Amazon Web Services latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: cloud/aws page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/aws.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/aws.adoc description: Find out about AWS components in Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-08-07" --- There are many components within Redpanda Connect which utilize AWS services. You will find that each of these components contains a configuration section under the field `credentials`, of the format: ```yml credentials: profile: "" id: "" secret: "" token: "" role: "" role_external_id: "" ``` This section contains many fields and it isn’t immediately clear which of them are compulsory and which aren’t. This document aims to make it clear what each field is responsible for and how it might be used. ## [](#credentials)Credentials The first thing to make clear is that _all_ of these fields are optional. When all fields are left blank Redpanda Connect will attempt to load credentials from a shared credentials file (`~/.aws/credentials`). The profile loaded will be `default` unless the `AWS_PROFILE` environment variable is set. ## [](#explicit-credentials)Explicit credentials By explicitly setting the credentials you are using at the component level it’s possible to connect to components using different accounts within the same Redpanda Connect process. ### [](#select-a-profile)Select a profile If you are using your shared credentials file but wish to explicitly select a profile set the `profile` field: ```yml credentials: profile: foo ``` ### [](#manual)Manual If you are using long term credentials for your account you only need to set the fields `id` and `secret`: ```yml credentials: id: foo # aws_access_key_id secret: bar # aws_secret_access_key ``` If you are using short term credentials then you will also need to set the field `token`: ```yml credentials: id: foo # aws_access_key_id secret: bar # aws_secret_access_key token: baz # aws_session_token ``` ## [](#assume-a-role)Assume a role It’s also possible to configure Redpanda Connect to [assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) using your credentials by setting the field `role` to your target role ARN. ```yml credentials: role: fooarn # Role ARN ``` This does NOT require explicit credentials, but it’s possible to use both. If you need to assume a role owned by another organization they might require you to [provide an external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html), in which case place it in the field `role_external_id`: ```yml credentials: role: fooarn # Role ARN role_external_id: bar_id ``` ## [](#related-guides)Related guides - [Authenticate to Amazon Aurora using IAM roles](https://docs.redpanda.com/connect/guides/cloud/aws-iam-aurora/) --- # Page 387: Google Cloud Platform **URL**: https://docs.redpanda.com/connect/guides/cloud/gcp.md --- # Google Cloud Platform --- title: Google Cloud Platform latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: cloud/gcp page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/gcp.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/gcp.adoc description: Find out about GCP components in Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- There are many components within Redpanda Connect which utilize Google Cloud Platform (GCP) services. You will find that each of these components require valid credentials. When running Redpanda Connect inside a Google Cloud environment that has a [default service account](https://cloud.google.com/iam/docs/service-accounts#default), it can automatically retrieve the service account credentials to call Google Cloud APIs through a library called Application Default Credentials (ADC). Otherwise, if your application runs outside Google Cloud environments that provide a default service account, you need to manually create one. Once you have a service account set up which has the required permissions, you can [create](https://console.cloud.google.com/apis/credentials/serviceaccountkey) a new Service Account Key and download it as a JSON file. Then all you need to do set the path to this JSON file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. Please refer to [this document](https://cloud.google.com/docs/authentication/production) for details. --- # Page 388: Message Delivery Semantics **URL**: https://docs.redpanda.com/connect/guides/delivery_semantics.md --- # Message Delivery Semantics --- title: Message Delivery Semantics latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: delivery_semantics page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: delivery_semantics.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/delivery_semantics.adoc description: Learn about Redpanda Connect's transactional model and error handling page-git-created-date: "2025-07-22" page-git-modified-date: "2025-12-17" --- Redpanda Connect guarantees strong message delivery semantics across its data pipelines by leveraging a transactional model centered around message batch acknowledgment. This ensures reliable, scalable, and predictable data movement between input and output systems, with particular support for **at-least-once delivery** without requiring intermediate message persistence. This topic describes how Redpanda Connect manages message delivery across the Input → Pipeline → Output architecture. ![delivery semantics arch](https://docs.redpanda.com/connect/shared/_images/delivery-semantics-arch.svg) ## [](#delivery-guarantees)Delivery guarantees An `ack()` function is associated with every batch of messages. This `ack()` function is a callback function provided by the input. It’s the output’s job to call the `ack()` function to inform the input of the success ("ack") or failure (negative ack or "nack") of the batch. The input can then decide what to do, for example, auto-retry nacks, propagate the nack upstream to input connectors that support it (Google PubSub), or drop the batch. When connecting to **at-least-once** inputs and outputs, Redpanda Connect guarantees **at-least-once** delivery without having to persist messages during transit. Key benefits: - Messages are retained unless configuration explicitly drops them - Every message is processed at-least-once, with optional retry logic for failures - Duplicates may occur during failure recovery, as is typical of at-least-once systems - Delivery status is communicated back to the input, enabling robust retry and propagation strategies - No need for disk-based checkpointing, reducing latency and infrastructure complexity ## [](#caveats-to-at-least-once)Caveats to at-least-once At-least-once delivery in a Redpanda Connect pipeline means every message is delivered at-least-once even if failures occur (which could result in some duplicates, but no lost data). Introducing a distributed or externally persisted state into a pipeline can compromise this guarantee. The pipeline’s built-in fault tolerance is based on the assumption that if an output fails, the message will be retried or reprocessed upon recovery. However, an external state (such as a remote cache or database used within the pipeline) may not reset on failure. It could retain an indication that a message was processed even if that message was never successfully processed by the output, and this mismatch can lead to message loss during recovery. For example, a pipeline uses a distributed cache to keep track of message IDs that have been seen in order to filter out duplicates. When a new message arrives, its ID is added to the cache immediately to mark it as processed. Now imagine that the pipeline’s output connector fails or the pipeline instance crashes just after updating the cache, but before the message actually reaches the destination. When the system comes back up, it will retry processing any unacknowledged messages, but because the cache still contains that message ID from the earlier attempt, the pipeline treats the message as a duplicate and drops it. As a result, that message is never delivered by the output at all, which is a direct violation of at-least-once guarantees. One way to mitigate this risk is to avoid using a state that outlives the pipeline process. For instance, keeping the cache in-memory (local to each pipeline instance) means it won’t preserve entries through a crash. If the process restarts, an in-memory cache would start fresh, allowing the system to reprocess any message that didn’t complete earlier instead of mistakenly skipping it. If you combine this with consistent partitioning of messages by key across horizontally scaled pipeline instances (so that each key is always handled by the same instance), you can still perform de-duplication within each partition while keeping state isolated per instance. In this setup, a failure that restarts one instance will clear that instance’s cache, and upon recovery it will reprocess its pending messages without interference from stale state. Even with such mitigations, if reliable delivery is paramount, a safer approach is to avoid relying on any external persistent state for eliminating duplicates within the pipeline. Instead, let the pipeline run with its native at-least-once behavior (allowing potential duplicates on retries) and design the downstream systems to handle duplicates idempotently. ## [](#error-handling)Error handling Redpanda Connect embraces a declarative and opt-in error-handling strategy: - A message error is represented as a flag, not a dropped message - If not explicitly handled in the pipeline, error messages flow through to the output stage - Error-handling components (for example, `try`, or `catch`) let you: - Retry failed operations - Route errors to a dead-letter queue - Fall back to alternate logic or output targets This model provides flexibility in building resilient pipelines without introducing unnecessary complexity. Refer to the specific [Error Handling](https://docs.redpanda.com/connect/configuration/error_handling/) documentation for further details and examples. --- # Page 389: Tune the Iceberg output for throughput **URL**: https://docs.redpanda.com/connect/guides/iceberg-output-tuning.md --- # Tune the Iceberg output for throughput --- title: Tune the Iceberg output for throughput latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: iceberg-output-tuning page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: iceberg-output-tuning.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/iceberg-output-tuning.adoc description: Configure the Redpanda Connect Iceberg output for high write throughput using tested recipes for append-only and keyed CDC workloads. page-topic-type: how-to personas: streaming_developer, data_engineer learning-objective-1: Configure an order-preserving Iceberg sink that sustains high throughput learning-objective-2: Configure an unordered Iceberg sink for maximum throughput learning-objective-3: Choose the recipe that matches your ordering and throughput requirements learning-objective-4: Tune a keyed change-data-capture sink that uses upsert or delete operations page-git-created-date: "2026-07-23" page-git-modified-date: "2026-08-26" --- Tune the [`iceberg` output](https://docs.redpanda.com/connect/components/outputs/iceberg/) for high write throughput to Apache Iceberg tables. Use these four tested recipes to choose the right trade-off between preserving message ordering, maximizing throughput, and applying keyed change-data-capture (CDC) operations. After reading this page, you will be able to: - Configure an order-preserving Iceberg sink that sustains high throughput - Configure an unordered Iceberg sink for maximum throughput - Choose the recipe that matches your ordering and throughput requirements - Tune a keyed change-data-capture sink that uses upsert or delete operations The two append recipes target roughly 10,000 rows or 10 seconds per commit, whichever comes first. This keeps commit sizes large enough to amortize the cost of each Iceberg snapshot while bounding commit latency. ## [](#prerequisites)Prerequisites - A configured Iceberg REST catalog and storage backend. See [the `iceberg` output reference](https://docs.redpanda.com/connect/components/outputs/iceberg/) for catalog and storage options. - Familiarity with [message batching](https://docs.redpanda.com/connect/configuration/batching/), [buffers](https://docs.redpanda.com/connect/components/buffers/about/), and the `max_in_flight` field. > ⚠️ **CAUTION** > > These recipes tune **append-only** sinks, where every message becomes a new row. They use a high `max_in_flight` to commit batches concurrently. Do not apply them to keyed change-data-capture workloads that use `row_operation` with `upsert` or `delete`. Keyed workloads require `max_in_flight: 1` to preserve per-key order, and their tuning levers are different: see [Recipe C](#tune-keyed-cdc-workloads). ## [](#recipe-a-preserve-message-ordering)Recipe A: preserve message ordering Use this recipe when you want to process records as a single ordered stream, rather than in parallel per partition. A single merged input stream preserves the processing order of records, because Redpanda Connect consumes the stream in order instead of parallelizing consumption per partition. To keep that ordered stream fast, decouple the input from the commit-bound output and batch at the output level: - A [`memory` buffer](https://docs.redpanda.com/connect/components/buffers/memory/) absorbs short input bursts and commit stalls, decoupling the input from commit latency. The buffer is bounded (500 MiB in this recipe): if the input sustains a higher rate than the sink can commit, the buffer fills and applies backpressure to the input. Size the buffer to your throughput multiplied by the commit interval. - Output-level `batching` feeds the Iceberg commit coalescer, so `max_in_flight` can dispatch concurrent writers. Without output batching, a single buffered stream starves the committer. In smoke tests at 1 vCPU, adding output batching raised throughput from 0.8 to 12.7 MB/s. ```yaml # Configure catalog, storage, namespace, and table as usual. buffer: memory: limit: 524288000 # 500 MiB: size to throughput x commit interval batch_policy: count: 10000 period: 10s output: iceberg: max_in_flight: 16 batching: count: 10000 period: 10s commit: max_snapshot_age: 24h ``` > 📝 **NOTE** > > `max_in_flight: 16` lets the output commit batches concurrently for throughput. Because concurrent batches can commit in any order, the order in which rows become visible in the Iceberg table is not guaranteed to match the input order, even though the input stream itself is processed in order. If you need commits applied in strict order, set `max_in_flight: 1`, at the cost of throughput. This recipe’s values are validated optima for this pipeline shape, so there is no need to raise them further. In benchmarks against an AWS Glue REST catalog, larger `max_in_flight` values and larger batch sizes produced the same mean throughput with a burstier delivery pattern, because Iceberg commits to a single table are applied sequentially. The recipe’s values give the smoothest commit cadence at full throughput. Batch cadences of 5 and 10 seconds performed identically. ## [](#recipe-b-maximize-throughput)Recipe B: maximize throughput Use this recipe when preserving processing order is not required, which is typical for append-only Iceberg sinks. This recipe reaches the highest throughput ceiling. Recipe B removes the buffer and parallelizes at the input instead. The [`redpanda` input](https://docs.redpanda.com/connect/components/inputs/redpanda/)'s `unordered_processing` processes each partition’s records in parallel and batches them in the read-ahead fetch buffer, feeding many concurrent commits to the coalescer. A higher `max_in_flight` of `32` lets those commits overlap. This configuration trades ordered processing for higher throughput. ```yaml input: redpanda: # Configure seed_brokers, topics, and consumer_group as usual. unordered_processing: enabled: true checkpoint_limit: 20000 batching: count: 10000 period: 10s output: iceberg: max_in_flight: 32 ``` > ❗ **IMPORTANT** > > `checkpoint_limit` must exceed `batching.count`, because it caps how many unacknowledged records a partition can hold in flight. If the limit is lower than the batch size (for example, the default `checkpoint_limit` of 1024 combined with `count: 10000`), batches can never fill to their target count. Every batch then flushes on the `period` timer at a fraction of the intended size, and each undersized batch becomes its own Iceberg commit. Set `checkpoint_limit` to at least twice `batching.count`. ## [](#compare-recipe-throughput)Compare recipe throughput The following table shows mean write throughput in MB/s for each append recipe as vCPUs scale. Keyed workloads follow a different cost model: see [Recipe C](#tune-keyed-cdc-workloads) and [Recipe D](#recipe-d). Results were measured on an Amazon Web Services (AWS) `c8g.4xlarge` instance against a roughly 178 GB dataset, using an AWS Glue REST catalog with Amazon S3 storage. | vCPU | Recipe A (ordered) | Recipe B (unordered) | | --- | --- | --- | | 1 | 15.9 | 38.3 | | 2 | 69.1 | 64.1 | | 4 | 114.2 | 98.6 | | 8 | 109.2 | 122.4 | When CPU is constrained, Recipe B’s per-partition parallelism gives a clear advantage. Recipe A scales strongly up to 4 vCPUs and then plateaus, while Recipe B continues to climb as more vCPUs become available. Your results depend on your catalog, storage backend, message size, and partition count. Use these numbers as a starting point and benchmark against your own workload. ## [](#choose-a-recipe)Choose a recipe | Requirement | Recipe | | --- | --- | | Ordered processing is not required (typical for append-only Iceberg sinks) | Recipe B: maximize throughput | | You want the stream processed in order | Recipe A: preserve message ordering | | The workload is keyed CDC (row_operation with upsert or delete) | Recipe C: keyed CDC with merge-on-read | | The workload is keyed CDC and the table must be readable by engines that cannot read equality deletes (for example, Snowflake or Databricks Unity Catalog) | Recipe D: keyed CDC with clustered-key copy-on-write | ## [](#tune-keyed-cdc-workloads)Recipe C: keyed CDC with merge-on-read Keyed workloads, where `row_operation` resolves to `upsert` or `delete`, follow a different cost model than the append recipes above, so they are tuned with different levers: - `max_in_flight` must be `1`. The output validates this at startup to guarantee last-writer-wins correctness: concurrent batches could otherwise commit out of order. See [Row-level operations](https://docs.redpanda.com/connect/components/outputs/iceberg/#row-level-operations). - Each batch that contains an `upsert` or `delete` becomes its own Iceberg snapshot. This one-to-one mapping is what guarantees correct results, so batch boundaries directly define snapshot boundaries. With one snapshot per batch and sequential commits, **batch size is the primary throughput lever**: throughput is the rows carried per batch divided by the time each commit takes. In benchmarks against an AWS Glue REST catalog, raising the batch size from 10,000 to 50,000 rows more than doubled measured upsert throughput. Size the batch against your latency budget and memory, and run regular snapshot expiry and compaction, because a keyed sink produces one snapshot per batch. ```yaml # Configure catalog, storage, namespace, and table as usual. buffer: memory: limit: 524288000 # 500 MiB: size to throughput x commit interval batch_policy: count: 50000 period: 10s output: iceberg: max_in_flight: 1 # required for keyed workloads row_operation: upsert # or an interpolated expression driven by the data identifier_fields: [ id ] batching: count: 50000 period: 10s commit: max_snapshot_age: 24h ``` ### [](#choose-a-merge-strategy)Choose a merge strategy The `merge_strategy` field controls how mutations are written, and it dominates keyed throughput: - `merge-on-read` (the default) writes small equality-delete files and sustains streaming rates: the numbers above were measured in this mode. Only engines that can read equality deletes can query the result. - `copy-on-write` rewrites every data file that contains an updated key, which makes the table readable by every engine. Its throughput is set by how the updated keys land across data files: keys arriving in contiguous runs sustained a rate comparable to merge-on-read in the same benchmark (see [Recipe D](#recipe-d)), while keys spread across the whole table sustained a small fraction of that, because every commit rewrites every file containing a touched key, and the scattered-key cost grows with table size. Use `merge-on-read` for continuous CDC streams whenever your query engines can read it. Use `copy-on-write` when they cannot, following [Recipe D](#recipe-d) and its key-clustering prerequisite. For the full decision guide and support matrix, see the `merge_strategy` field in the [`iceberg` output reference](https://docs.redpanda.com/connect/components/outputs/iceberg/#merge-strategies). ## [](#recipe-d)Recipe D: keyed CDC with clustered-key copy-on-write Use this recipe when the table must be readable by engines that cannot read equality deletes, such as Snowflake or the Databricks Unity Catalog, so `merge-on-read` is not an option. Copy-on-write rewrites every data file that contains an updated key, so the write cost per batch is set by how many files the batch’s keys touch. That makes key arrival order the decisive factor: - **Clustered keys** (arriving in contiguous runs) concentrate each batch’s updates into the few files covering that key range. Measured with the Recipe C configuration plus `merge_strategy: copy-on-write` against an AWS Glue REST catalog, this sustained a steady rate in the same range as merge-on-read on the identical workload, with no degradation over time. - **Scattered keys** spread each batch’s updates across many files, so each commit rewrites a large share of the table. This sustains a small fraction of the clustered rate, and that share grows with table size. This is an inherent property of the copy-on-write format, not a tunable. Address scattered keys upstream, in the data layout, as described below. The configuration is Recipe C with one change: ```yaml # Configure catalog, storage, namespace, and table as usual. buffer: memory: limit: 524288000 batch_policy: count: 50000 period: 10s output: iceberg: max_in_flight: 1 # required for keyed workloads row_operation: upsert identifier_fields: [ id ] merge_strategy: copy-on-write batching: count: 50000 period: 10s commit: max_snapshot_age: 24h ``` To satisfy the clustering prerequisite when your source does not already produce keys in runs, arrange it upstream: order the capture by primary key where the source supports it, or partition the source topic by key range so each partition carries a contiguous slice of the key space. If the keys cannot be clustered, expect scattered-key performance and treat the sink as batch or low-churn only. Plan for regular table maintenance with this recipe: every mutating batch rewrites files and adds a snapshot, so schedule compaction, snapshot expiry, and orphan-file removal as part of normal operations. See the `merge_strategy` field in the [`iceberg` output reference](https://docs.redpanda.com/connect/components/outputs/iceberg/#merge-strategies) for the maintenance guidance and support matrix. ## [](#next-steps)Next steps - [Performance tuning](https://docs.redpanda.com/connect/guides/performance_tuning/) - [Message batching](https://docs.redpanda.com/connect/configuration/batching/) - [iceberg output reference](https://docs.redpanda.com/connect/components/outputs/iceberg/) - [Merge strategies](https://docs.redpanda.com/connect/components/outputs/iceberg/#merge-strategies), for the write-cost trade-offs of `merge_strategy` on keyed workloads --- # Page 390: Migration **URL**: https://docs.redpanda.com/connect/guides/migration.md --- # Migration --- title: Migration latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: migration/index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/index.adoc description: Migrate between different Redpanda Connect versions, including preparation steps and best practices. page-git-created-date: "2024-06-03" page-git-modified-date: "2026-01-27" --- > ❗ **IMPORTANT** > > This section is related to migrating to a newer version of Redpanda Connect. For information about migrating your data using Redpanda Migrator, see [Redpanda Migrator](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/). - [Migrate to the Unified Redpanda Migrator](migrate-unified-redpanda-migrator/) Learn how to migrate from legacy migrator components to the unified `redpanda_migrator` input/output pair in Redpanda Connect 4.67.5+. - [Migrating to Version 4](v4/) Learn how to migrate to Redpanda Connect v4, with insights into new functionalities and migration tips. - [Migrating to Version 3](v3/) Learn how to migrate to Redpanda Connect v3. - [Migrating to Version 2](v2/) Learn how to migrate to Redpanda Connect v2. --- # Page 391: Migrate to the Unified Redpanda Migrator **URL**: https://docs.redpanda.com/connect/guides/migration/migrate-unified-redpanda-migrator.md --- # Migrate to the Unified Redpanda Migrator --- title: Migrate to the Unified Redpanda Migrator latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: migration/migrate-unified-redpanda-migrator page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/migrate-unified-redpanda-migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/migrate-unified-redpanda-migrator.adoc description: Learn how to migrate from legacy migrator components to the unified redpanda_migrator input/output pair in Redpanda Connect 4.67.5+. page-git-created-date: "2025-10-24" page-git-modified-date: "2026-10-10" --- > ❗ **IMPORTANT** > > This page is about migrating to a newer version of Redpanda Connect. For information about migrating your data using Redpanda Migrator, see [Redpanda Migrator](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/). This guide explains how to migrate from legacy migrator components (`redpanda_migrator_bundle`, `legacy_redpanda_migrator` and `legacy_redpanda_migrator_offsets`) to the unified `redpanda_migrator` input/output pair introduced in Redpanda Connect 4.67.5+. The unified migrator consolidates all migration logic into a single input/output pair, simplifying configuration and improving reliability. ## [](#overview)Overview | Available in | Redpanda Connect 4.67.5+ | | --- | --- | | Legacy status | Deprecated in 4.67.5, removed in 4.85.0 | | Compatibility | Not backward-compatible | | Configuration model | One input and one output, paired by label | | Primary control | All migration logic resides in the output component | Key concepts: - Components are paired by matching `label` values. - The input defines the source cluster and schema registry. - The output defines the destination cluster, schema registry, and migration behavior. - Topic mapping and consumer group migration are configured in the output. ## [](#architectural-changes)Architectural changes ### [](#legacy-architecture)Legacy architecture A complex bundle (`redpanda_migrator_bundle`) that managed three subcomponents: - `redpanda_migrator`: Data transfer - `schema_registry`: Schema synchronization - `redpanda_migrator_offsets`: Consumer group offsets This design required complex internal routing and sequencing. ### [](#unified-architecture)Unified architecture A single `redpanda_migrator` input/output pair replaces the bundle: - **Input**: Consumes from the source Kafka cluster. - **Output**: Handles topic creation, schema synchronization, ACLs, and consumer group offsets. Benefits: - Simplified setup: all configuration consolidated in one output component. - Improved coordination: no internal routing or wrapper logic. - Enhanced control: fine-grained schema and topic options, improved offset handling. ## [](#migration-steps)Migration steps Follow this checklist in order to ensure a safe, low-risk migration. - Back up your existing configurations. - Add new `input.redpanda_migrator` and `output.redpanda_migrator` components with matching labels. - Move source Kafka and Schema Registry settings to the input. - Move destination Kafka and Schema Registry settings to the output. - Replace `topic_prefix` with `topic` using interpolation syntax. - Move offset settings to `output.redpanda_migrator.consumer_groups`. - Remove deprecated fields. - Validate configuration with `rpk connect lint`. - Test using non-production topics first. - Monitor logs and performance during migration. - Remove legacy configuration after successful migration. ## [](#field-mapping-reference)Field mapping reference ### [](#bundle-wrapper-redpanda_migrator_bundle)Bundle wrapper (`redpanda_migrator_bundle`) #### [](#input-mapping)Input mapping | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator | input.redpanda_migrator | Moved | Source cluster connection | | schema_registry | input.redpanda_migrator.schema_registry | Moved | Source schema registry | | migrate_schemas_before_data | - | Removed | Controlled by output schema interval | | consumer_group_offsets_poll_interval | output.redpanda_migrator.consumer_groups.interval | Moved | Now controls sync frequency | #### [](#output-mapping)Output mapping | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator | output.redpanda_migrator | Moved | Destination cluster configuration | | schema_registry | output.redpanda_migrator.schema_registry | Moved | Destination schema registry | | translate_schema_ids | output.redpanda_migrator.schema_registry.translate_ids | Moved | Schema ID translation | | input_bundle_label | label | Replaced | Input and output paired by label | ### [](#data-migration-fields)Data migration fields | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | All (*) | input.redpanda_migrator.* | Moved | Direct mapping | | topics (explicit list) | input.redpanda_migrator.topics | Unchanged | Still supported for explicit lists | | regexp_topics: true | input.redpanda_migrator.regexp_topics_include, regexp_topics_exclude | Deprecated | Use include/exclude arrays for pattern-based selection | | topic_prefix | output.redpanda_migrator.topic | Replaced | Use interpolation, for example 'prefix_${! @kafka_topic }' | | replication_factor_override, replication_factor | output.redpanda_migrator.topic_replication_factor | Replaced | Unified field | | input_resource | label | Replaced | Label pairing replaces internal routing | | - | output.redpanda_migrator.provenance_header | New | Optional header for tracking message source cluster | ### [](#schema-migration-fields)Schema migration fields | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | Connection fields | input.redpanda_migrator.schema_registry.* | Moved | Source schema registry | | subject_filter | output.redpanda_migrator.schema_registry.include, exclude | Replaced | Use regex lists for filtering | | include_deleted | output.redpanda_migrator.schema_registry.include_deleted | Moved | Configured on destination | | backfill_dependencies | output.redpanda_migrator.schema_registry.versions | Replaced | Choose all or latest | ### [](#consumer-group-offset-migration)Consumer group offset migration The `redpanda_migrator_offsets` pair is replaced by the `consumer_groups` block in the output. | Legacy Component | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator_offsets (input/output) | output.redpanda_migrator.consumer_groups | Replaced | Unified control block | ## [](#migration-example)Migration example The following example demonstrates a complete migration from legacy to unified components. Legacy configuration ```yaml input: label: "source_cluster" redpanda_migrator_bundle: legacy_redpanda_migrator: seed_brokers: [ "source-kafka:9092" ] topics: [ "orders", "payments" ] consumer_group: "migration_group" schema_registry: url: "http://source-registry:8081" migrate_schemas_before_data: false consumer_group_offsets_poll_interval: 30s output: redpanda_migrator_bundle: legacy_redpanda_migrator: seed_brokers: [ "destination-redpanda:9092" ] topic_prefix: "migrated_" schema_registry: url: "http://destination-registry:8081" translate_schema_ids: true input_bundle_label: "source_cluster" ``` Unified configuration ```yaml input: label: "migration_pipeline" (1) redpanda_migrator: # Source Kafka settings seed_brokers: [ "source-kafka:9092" ] # Pattern-based topic selection (for migrating all topics except system topics) # Note: You can still use explicit lists: topics: [ "orders", "payments" ] regexp_topics_include: [ '.' ] (2) regexp_topics_exclude: [ '^_' ] (3) consumer_group: "migration_group" # Source Schema Registry settings schema_registry: url: "http://source-registry:8081" output: label: "migration_pipeline" (4) redpanda_migrator: # Destination Redpanda settings seed_brokers: [ "destination-redpanda:9092" ] # Topic mapping (replaces topic_prefix) topic: 'migrated_${! @kafka_topic }' (5) # Add source cluster tracking header provenance_header: "x-source-cluster" (6) # Destination Schema Registry and migration settings schema_registry: url: "http://destination-registry:8081" translate_ids: true # Rename subjects subject: 'migrated_${! metadata("schema_registry_subject") }' # Consumer group migration settings consumer_groups: enabled: true interval: 30s (7) ``` | 1 | Labels are now used for pairing input and output. | | --- | --- | | 2 | Match all topics using regex pattern. | | 3 | Exclude internal/system topics starting with underscore. | | 4 | Matching label pairs the input and output components. | | 5 | Use interpolation syntax to replicate topic_prefix behavior. | | 6 | Adds a header to track which cluster messages originated from, useful for debugging and auditing. | | 7 | Replaces consumer_group_offsets_poll_interval. | ## [](#validation)Validation Before running, validate your configuration: ```bash rpk connect lint config.yaml ``` Then test on a small set of topics before running full migrations. ## [](#troubleshooting)Troubleshooting | Problem | Likely Cause | Solution | | --- | --- | --- | | Labels do not match | Input and output labels differ | Use identical, case-sensitive labels. | | Topic interpolation errors | Incorrect syntax | Use topic: 'prefix_${! @kafka_topic }' with quotes and !. | | Schema registry connection fails | Incorrect registry placement | The source registry must be in the input. The destination registry must be in the output. | | Consumer group migration not working | Missing consumer_groups.enabled: true | Ensure consumer group migration is explicitly enabled. | ## [](#after-migration)After migration After verifying that the new migrator works as expected: - Remove legacy configuration files. - Update internal documentation and runbooks. - Train your team on the new configuration model. - See the [`redpanda_migrator` output](https://docs.redpanda.com/connect/components/outputs/redpanda_migrator/) reference for advanced configuration options. --- # Page 392: Migrating to Version 2 **URL**: https://docs.redpanda.com/connect/guides/migration/v2.md --- # Migrating to Version 2 --- title: Migrating to Version 2 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: migration/v2 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/v2.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/v2.adoc description: Learn how to migrate to Redpanda Connect v2. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-14" --- Redpanda Connect version 2 comes with a small number of backwards incompatible changes that are organized into three sections; [configuration](#configuration), [service](#service) and [Go API](#go-api). These changes are considered minor and therefore upgrading is not expected to cause significant problems for any users. ## [](#configuration)Configuration ### [](#type-inference)Type inference Version 2 comes with the ability to infer the `type` of components in configuration files whenever the field is omitted. You can read more about this behavior [here](https://docs.redpanda.com/connect/configuration/about/). This feature is not expected to impact the vast majority of users. However, there is one exception where a malformed section containing unused type parameters but a missing `type` field will be interpreted differently. For example, the following config: ```yml pipeline: processors: - # type: text text: operator: set value: "delete all your content" ``` In V1 would be interpreted as a `bounds_check` processor as it is the default processor type, whereas V2 would infer this to be a `text` processor based on its fields. #### [](#migration-guide)Migration guide Most users should not be impacted by this change, and a config file that is vulnerable to the regression would report linting errors in V1. You can quickly verify that your configs are interpreted without regression by comparing the output of `benthos -c ./yourconfig.yaml --print-yaml` with V1 and V2. If they are the same then you are not affected. ### [](#field-default-value-changes)Field default value changes In version 2 the field `unsubscribe_on_close` of the `nats_stream` input is now `false` by default. ## [](#service)Service The recommended way to create plugins for Redpanda Connect is outlined in [this repository](https://github.com/benthosdev/benthos-plugin-example). Therefore the following experimental plugin related flags have been removed from the service: `swap-envs`, `plugins-dir`, `list-input-plugins`, `list-output-plugins`, `list-processor-plugins`, `list-condition-plugins`. The flag `swap-envs` has also been removed for clarity, as it had no impact on JSON reference resolution. If this flag is being used please open an issue and it can be reimplemented to be fully compliant. ## [](#go-api)Go API ### [](#condition-package-moved)Condition package moved The package `github.com/Jeffail/benthos/lib/processor/condition` has been changed to `github.com/Jeffail/benthos/lib/condition`. Migrating should be a simple case of applying a find/replace on your codebase: ```sh find . -name "*.go" | \ xargs sed -i 's/benthos\/lib\/processor\/condition/benthos\/lib\/condition/g' ``` ### [](#interface-changes)Interface changes The following interface changes have occurred to core Redpanda Connect components: - `types.Cache` now has `types.Closable` embedded. - `types.RateLimit` now has `types.Closable` embedded. - `types.Manager` has new method `GetPlugin`. - `log.Modular` has new method `WithFields`. --- # Page 393: Migrating to Version 3 **URL**: https://docs.redpanda.com/connect/guides/migration/v3.md --- # Migrating to Version 3 --- title: Migrating to Version 3 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: migration/v3 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/v3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/v3.adoc description: Learn how to migrate to Redpanda Connect v3. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-14" --- Redpanda Connect version 3 comes with some breaking [service](#service) and [configuration](#configuration) changes as well as a few breaking [API changes](#go-api). ## [](#service)Service ### [](#memory-map-file-buffer-removed)Memory map file buffer removed The long deprecated `mmap_file` buffer has been removed. If you were still relying on this buffer implementation then please [raise an issue](https://github.com/redpanda-data/connect/issues). ### [](#old-metrics-paths-removed)Old metrics paths removed The following undocumented metrics paths have been removed: - `input.parts.count` - `input.read.success` - `input.read.error` - `input.send.success` - `input.send.error` - `input.ack.success` - `input.ack.error` - `output.running` - `output.parts.count` - `output.send.success` - `output.parts.send.success` - `output.send.error` All of these paths have [remaining equivalents](https://docs.redpanda.com/connect/components/metrics/about/). ### [](#metric-path-changes)Metric path changes The `http_client` output client metrics have been renamed from `output.**.output.http_client**` **to `output.`**`.client`. ## [](#configuration)Configuration ### [](#metrics-prefix)Metrics prefix The configuration field `prefix` within `metrics` has been moved from the root of the config object to individual types. E.g. when using `statsd` the field `metrics.prefix` should be replaced with `metrics.statsd.prefix`. ### [](#json-paths)JSON paths Many components within Redpanda Connect use an unspecified "JSON dot path" syntax for querying and setting fields within JSON documents. The format of these paths has been formalized to make them clearer and more generally useful, but this potentially breaks your paths when they query against hierarchies that contain arrays. The formal specification for v3 can be found [in this document](https://docs.redpanda.com/connect/configuration/field_paths/). The following components are affected: - `awk` processor (all of the `json_*` functions) - `json` processor (`path` field) - `process_field` processor (`path` field) - `process_map` processor (`premap`, `premap_optional`, `postmap` and `postmap_optional` fields) - `check_field` condition (`path` field) - `json_field` function interpolation - `s3` input (`sqs_body_path`, `sqs_bucket_path` and `sqs_envelope_path` fields) - `dynamodb` output (`json_map_columns` field values) #### [](#migration-guide)Migration guide In order to replicate the exact same behavior as currently exists your paths should be updated to include the character `*` wherever an array exists. For example, the default value of `sqs_body_path` for the `s3` input has been updated from `Records.s3.object.key` to `Records.*.s3.object.key`. ### [](#process-dag-stage-names)Process DAG stage names The `process_dag` processor now only permits workflow stage names matching the following regular expression: `[a-zA-Z0-9_-]+`. The reasoning for this restriction is to potentially expand the features of `process_dag` in the future with custom root fields (e.g. `$on_error`). ## [](#go-api)Go API ### [](#modules)Modules Redpanda Connect now fully adheres to [Go Modules](https://github.com/golang/go/wiki/Modules), import paths must therefore now contain the major version (v3) like so: ```go import "github.com/Jeffail/benthos/v3/lib/processor" ``` It should be pretty quick to update your imports, either using a tool or just: ```sh grep "Jeffail/benthos" . -Rl | grep -e "\.go$" | xargs -I{} sed -i 's/Jeffail\/benthos/Jeffail\/benthos\/v3/g' {} ``` ### [](#other)Other - Constructors for buffer components now require a `types.Manager`, giving them parity with other components: `buffer.New(conf Config, mgr types.Manager, log log.Modular, stats metrics.Type) (Type, error)` --- # Page 394: Migrating to Version 4 **URL**: https://docs.redpanda.com/connect/guides/migration/v4.md --- # Migrating to Version 4 --- title: Migrating to Version 4 latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: migration/v4 page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/v4.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/v4.adoc description: Learn how to migrate to Redpanda Connect v4, with insights into new functionalities and migration tips. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Redpanda Connect has been at major version 3 [for more than two years](https://docs.redpanda.com/blog/2021/01/04/v4-roadmap), during which time it has gained a huge amount of functionality without introducing any breaking changes. However, the number of components, APIs and features that have been deprecated in favour of better solutions has grown steadily and the time has finally come to purge them. There are also some areas of functionality that have been improved with breaking changes. This document outlines the changes made to Redpanda Connect since V3 and tips for how to migrate to V4 in places where those changes are significant. ## [](#deprecated-components-removed)Deprecated components removed All components, features and configuration fields that were marked as deprecated in the latest release of V3 have been removed in V4. In order to detect deprecated components or fields within your existing configuration files you can run the linter from a later release of V3 Redpanda Connect with the `--deprecated` flag: ```sh benthos lint --deprecated ./configs/*.yaml ``` This should report all remaining deprecated components. All deprecated components have favoured alternative solutions in V3, so it should be possible to slowly eliminate deprecated aspects of your config using V3 before upgrading. ### [](#unit-test-directories)Unit test directories The `benthos test` subcommand no longer walks paths when they are directories. Instead use explicit triple-dot syntax (`./dir/...`) or wildcard patterns. ## [](#new-go-module-name)New Go module name For users of the Go plugin APIs the import path of this module needs to be updated to `github.com/benthosdev/benthos/v4`, like so: ```go import "github.com/benthosdev/benthos/v4/public/service" ``` ## [](#pulsar-components-disabled-for-now)Pulsar components disabled (for now) There have been multiple issues with the Go Pulsar client libraries. Since some are still outstanding and causing problems with unrelated components the decision has been made to remove the `pulsar` input and output from standard builds. However, it is still possible to build custom versions of Redpanda Connect with them included by importing the package `./public/components/pulsar`: ```go package main import ( "context" "github.com/Jeffail/benthos/v3/public/service" // Import all plugins defined within the repo. _ "github.com/benthosdev/benthos/v4/public/components/all" _ "github.com/benthosdev/benthos/v4/public/components/pulsar" ) func main() { service.RunCLI(context.Background()) } ``` ## [](#pipeline-threads-behavior-change)Pipeline threads behavior change In V3 the `pipeline.threads` field defaults to 1. If this field is explicitly set to `0` it will automatically match the number of CPUs on the host machine. In V4 this will change so that the default value of `pipeline.threads` is `-1`, where this value indicates we should match the number of host CPUs. An explicit value of `0` is still considered valid and functionally equivalent to `-1`. [Related issue](https://github.com/redpanda-data/connect/issues/399). ## [](#old-style-interpolation-functions-removed)Old Style Interpolation Functions Removed The original style of interpolation functions, where you specify a function name followed by a colon and then any arguments (`${!json:foo,1}`) has been deprecated (and undocumented) for a while now. What we’ve had instead is a subset of Bloblang allowing you to use functions directly (`${! json("foo").from(1) }`), but with the old style still supported for backwards compatibility. However, supporting the old style means our parsing capabilities are weakened and so it is now removed in order to allow more powerful interpolations in the future. ## [](#bloblang-changes)Bloblang changes The functions `meta`, `root_meta`, `error` and `env` now return `null` when the target value does not exist. This is in order to improve consistency across different functions and query types. In cases where a default empty string is preferred you can add `.or("")` onto the function. In cases where you want to throw an error when the value does not exist you can add `.not_null()` onto the function. [Related issue](https://github.com/redpanda-data/connect/issues/571). ### [](#root-referencing)Root referencing It is now possible to reference the `root` of the document being created within a mapping query, i.e. `root.hash = root.string().hash("xxhash64")`. ## [](#env-var-docker-configuration)Env var Docker configuration Docker builds will no longer come with a default config that contains generated environment variables. This system doesn’t scale at all for complex configuration files and was becoming a challenge to maintain (and also huge). Instead, the new `-s` flag has been the preferred way to configure Redpanda Connect through arguments and will need to be used exclusively in V4. It’s worth noting that this does not prevent you from defining your own env var based configuration and adding that to your docker image. It’s entirely possible to copy the config from V3 and have that work, it just won’t be present by default any more. In order to migrate to the `-s` flag use the path of the fields you’re setting instead of the generated environment variables, so: ```sh docker run --rm -p 4195:4195 jeffail/benthos \ -e "INPUT_TYPE=http_server" \ -e "OUTPUT_TYPE=kafka" \ -e "OUTPUT_KAFKA_ADDRESSES=kafka-server:9092" \ -e "OUTPUT_KAFKA_TOPIC=benthos_topic" ``` Becomes: ```sh docker run --rm -p 4195:4195 jeffail/benthos \ -s "input.type=http_server" \ -s "output.type=kafka" \ -s "output.kafka.addresses=kafka-server:9092" \ -s "output.kafka.topic=benthos_topic" ``` ## [](#old-plugin-apis-removed)Old plugin APIs removed Any packages from within the `lib` directory have been removed. Please use only the APIs within the `public` directory, the API docs count be found on [pkg.go.dev](https://pkg.go.dev/github.com/benthosdev/benthos/v4/public), and examples can be found in the [`benthos-plugin-example` repository](https://github.com/benthosdev/benthos-plugin-example). These new APIs can be found in V3 so if you have many components you can migrate them incrementally by sticking with V3 until completion. Many of the old packages within `lib` can also still be found within `internal`, if you’re in a pickle you can find some of those APIs and copy/paste them into your own repository. ## [](#caches)Caches All caches that support retries have had their retry/backoff configuration fields modified in order to be more consistent. The new common format is: ```yml retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` In cases where it might be desirable to disable retries altogether (the `ristretto` cache) there is also an `enabled` field. ### [](#ttl-changes)TTL changes Caches that support TTLs have had their `ttl` fields renamed to `default_ttl` in order to make it clearer that their purpose is to provide a fallback. All of these values are now duration string types, i.e. a cache with an integer seconds based field with a previous value of `60` should now be defined as `60s`. ## [](#field-default-changes)Field default changes Lots of fields have had default values removed in cases where they were deemed unlikely to be useful and likely to cause frustration. This specifically applies to any `url`, `urls`, `address` or `addresses` fields that may have once had a default value containing a common example for the particular service. In most cases this should cause minimal disruption as the field is non-optional and therefore not specifying it explicitly will result in config errors. [Related issue](https://github.com/redpanda-data/connect/issues/392). However, there are the following exceptions that are worth noting: ### [](#the-http-processor-and-http_client-output-no-longer-create-multipart-requests-by-default)The `http` processor and `http_client` output no longer create multipart requests by default The `http` processor and `http_client` output now execute message batch requests as individual requests by default. This behavior can be disabled by explicitly setting `batch_as_multipart` to `true`. ### [](#output-lines-codec-no-longer-adds-extra-batch-newlines)Output `lines` codec no longer adds extra batch newlines All outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the `lines` codec (`socket`, `stdout`, `file`, `sftp`) no longer do this by default. This was originally kept for backwards compatibility but was often seen as an unexpected and annoying behavior. It is still possible to add these end-of-batch newlines in a more consistent way by either adding an empty message to the end of batches, or by adding a newline to the last message of the batch. ### [](#the-switch-output-retry_until_success)The `switch` output `retry_until_success` By default the `switch` output continues retrying switch case outputs until success. This default was sensible at the time as we didn’t have a concept of intentionally nacking messages, and therefore a nacked message was likely a recoverable problem and retrying internally means that messages matching multiple cases wouldn’t produce duplicates. However, since then Redpanda Connect has evolved and a very common pattern with the `switch` output is to reject messages that failed during processing using the `reject` output. But because of the default value of `retry_until_success` many users end up in a confusing situation where using a `reject` output results in the pipeline blocking indefinitely until they discover this field. Therefore the default value of `retry_until_success` will now be `false`, which means users that aren’t using a `reject` flow in one of their switch cases, and have a configuration where messages could match multiple cases, should explicitly set this field to `true` in order to avoid potential duplicates during downstream outages. ### [](#aws-region-fields)AWS `region` fields Any configuration sections containing AWS fields no longer have a default `region` of `eu-west-1`. Instead, the field will be empty by default, where unless explicitly set the environment variable `AWS_REGION` will be used. This will cause problems for users where they expect the region `eu-west-1` to be targeted when neither the field nor the environment variable `AWS_REGION` are set. [Related issue](https://github.com/redpanda-data/connect/issues/696). ## [](#clickhouse-driver-changes)Clickhouse driver changes The `clickhouse` SQL driver Data Source Name format parameters have been changed due to a client library update (details can be found at [https://github.com/ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go)). A compatibility layer has been added that makes a best attempt to translate the old DSN format to the new one, but some parameters may not carry over exactly. This update also means placeholders in `sql_raw` queries should be in dollar syntax. ## [](#serverless-default-output)Serverless default output The default output of the serverless distribution of Redpanda Connect is now the following config: ```yml output: switch: retry_until_success: false cases: - check: errored() output: reject: "processing failed due to: ${! error() }" - output: sync_response: {} ``` This change was made in order to return processing errors directly to the invoker by default. ## [](#metrics-changes)Metrics changes The metrics produced by a Redpanda Connect stream have been greatly simplified and now make better use of labels/tags in order to provide component-specific insights. The configuration and behavior of metrics types has also been made more consistent, with metric names being the same throughout and `mapping` now being a general top-level field. For a full overview of the new system check out the [metrics about page](https://docs.redpanda.com/connect/components/metrics/about/). [Related issue](https://github.com/redpanda-data/connect/issues/1066). ### [](#field-prefix-is-gone)Field `prefix` is gone Some metrics components such as `prometheus` had a `prefix` field for setting a prefix to all metric names. These fields are now gone, if you want to reintroduce these prefixes you can use the general purpose `mapping` field. For example, if we previously had a config: ```yml metrics: prometheus: prefix: ${METRICS_PREFIX:benthos} ``` We need to delete that prefix and add a mapping that renames metric names: ```yaml metrics: mapping: 'root = env("METRICS_PREFIX").or("benthos") + "_" + this' prometheus: {} ``` ### [](#the-http_server-type-renamed-to-json_api)The `http_server` type renamed to `json_api` The name given to the generic JSON API metrics type was `http_server`, which was confusing as it isn’t the only metrics output type that presents as an HTTP server endpoint. This type was also only originally intended for local debugging, which the `prometheus` type is also good for. In order to distinguish this metrics type by its unique feature, which is that it exposes metrics as a JSON object, it has been renamed to `json_api`. ### [](#the-stdout-type-renamed-to-logger)The `stdout` type renamed to `logger` The `stdout` metrics type now emits metrics using the Redpanda Connect logger, and therefore also matches the logger format. As such, it has been renamed to `logger` in order to reflect that. ### [](#no-more-dots)No more dots In V3 metrics names contained dots in order to represent pseudo-paths of the source component. In V4 all metric names produced by Redpanda Connect have been changed to contain only alpha-numeric characters and underscores. It is recommended that any custom metric names produced by your `metric` processors and custom plugins should match this new format for consistency. Since dots were invalid characters in Prometheus metric names, in V3 the `prometheus` metrics type made some automatic modifications to all names before registering them. This rewrite first replaced all `-` and _characters to a double underscore (`_`_), and then replaced all `.` characters with `_`. This was an ugly work around and has been removed in V4, but means in previous cases where custom metrics containing dots were automatically converted you will instead see error logs reporting that the names were invalid and therefore ignored. If you wish to retain the old rewrite behavior you can reproduce it with the new `mapping` field: ```yml metrics: mapping: 'root = this.replace("_", "__").replace("-", "__").replace(".", "_")' prometheus: {} ``` However, it’s recommended to change your metric names instead. ## [](#tracing-changes)Tracing changes Distributed tracing within Redpanda Connect is now done via the Open Telemetry client library. Unfortunately, this client library does not support the full breadth of options as we had before. As such, the `jaeger` tracing type now only supports the `const` sampling type, and the field `service_name` has been removed. This will likely mean tracing output will appear different in this release, and if you were relying on code that extracts and interacts with spans from messages in your custom plugins then it will need to be converted to use the official Open Telemetry APIs. [Related issue](https://github.com/redpanda-data/connect/issues/872). ## [](#logging-changes)Logging changes The `logger` config section has been simplified, the new default set to `logfmt`, and the `classic` format removed. The default value of `add_timestamp` has also been changed to `false`. [Related issue](https://github.com/redpanda-data/connect/issues/589). ## [](#automatic-max-in-flight)Automatic max In flight Outputs that compose other outputs (`broker`, `switch`, etc) no longer require their own `max_in_flight` settings as they will automatically saturate their composed outputs. This includes outputs that compose resources. ## [](#processor-batch-behavior-changes)Processor batch behavior changes Some processors that once executed only once per batch have been updated to execute upon each message individually by default. This change has been made because it was felt the individual message case was considerably more common (and intuitive) and that it is possible to satisfy the batch-wide behavior in other ways that are opt-in, such as by placing the processors within a `branch` and having your `request_map` explicit for a single `batch_index` (i.e. `request_map: root = if batch_index() != 0 { deleted() }`). [Related issue](https://github.com/redpanda-data/connect/issues/408). ### [](#processor-parts-field-removed)Processor `parts` field removed Many processors previously had a `parts` field, which allowed you to explicitly list the indexes of a batch to apply the processor to. This field had confusing naming and was rarely used (or even known about). Since that same behavior can be reproduced by placing the processor within a `branch` (or `switch`) all `parts` fields have been removed. ### [](#dedupe)`dedupe` The `dedupe` processor has been reworked so that it now acts upon individual messages by default. It’s now mandatory to specify a `key`, and the `parts` and `hash` fields have been removed. Instead, specify full-content hashing with interpolation functions in the `key` field, e.g. `${! content().hash("xxhash64") }`. In order to deduplicate an entire batch it is likely easier to use a `cache` processor with the `add` operator: ```yml pipeline: processors: # Try and add one message to a cache that identifies the whole batch - branch: request_map: | root = if batch_index() == 0 { this.id } else { deleted() } processors: - cache: operator: add key: ${! content() } value: t # Delete all messages if we failed - mapping: | root = if errored().from(0) { deleted() } ``` ### [](#log)`log` The `log` processor now executes for every message of batches by default. ### [](#sleep)`sleep` The `sleep` processor now executes for every message of batches by default. ## [](#broker-ditto-macro-gone)Broker ditto macro gone The hidden macro `ditto` for broker configs is now removed. Use the `copies` field instead. For some edge cases where `copies` does not satisfy your requirements you may be better served using [configuration templates](https://docs.redpanda.com/connect/configuration/templating/). If all else fails then please [reach out](https://redpanda.com/slack) and we can look into other solutions. --- # Page 395: Monitor Pipeline Status **URL**: https://docs.redpanda.com/connect/guides/monitor-pipeline-status.md --- # Monitor Pipeline Status --- title: Monitor Pipeline Status latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: monitor-pipeline-status page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: monitor-pipeline-status.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/monitor-pipeline-status.adoc description: Monitor Redpanda Connect pipeline health using status events, set up alerts for connection failures, and troubleshoot common issues. learning-objective-1: Consume status events from a Kafka topic learning-objective-2: Configure alerts for pipeline failures and connection errors learning-objective-3: Troubleshoot common status reporting issues page-git-created-date: "2026-06-16" page-git-modified-date: "2026-06-16" --- > 📝 **NOTE** > > This component requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. Monitor your Redpanda Connect pipelines in real-time by consuming status events. Status events help you detect connection failures, track pipeline health, and troubleshoot issues before they impact data processing. After reading this page, you will be able to: - Consume status events from a Kafka topic - Configure alerts for pipeline failures and connection errors - Troubleshoot common status reporting issues For status event type definitions and message format, see [`status_topic` configuration reference](https://docs.redpanda.com/connect/components/redpanda/about/#status_topic). ## [](#prerequisites)Prerequisites - Configure the [`redpanda` service](https://docs.redpanda.com/connect/components/redpanda/about/) with `status_topic` enabled - Ensure you have a valid Enterprise Edition license - status reporting is an enterprise feature - Have access to consume from the status topic ## [](#consume-status-events)Consume status events Use `rpk` to consume status events in real-time: ```bash rpk topic consume __redpanda.connect.status --format json ``` Or use any Kafka consumer configured to read from the status topic. ## [](#understand-the-pipeline-lifecycle)Understand the pipeline lifecycle Status events follow this lifecycle: | Event type | Description | | --- | --- | | Startup: TYPE_INITIALIZING | Pipeline parsed configuration and is starting | | Running: TYPE_CONNECTION_HEALTHY | Heartbeat every 30 seconds when connections are healthy | | Errors: TYPE_CONNECTION_ERROR | One or more connections have failed (heartbeat every 30 seconds) | | Shutdown: TYPE_EXITING | Pipeline is stopping (gracefully or due to error) | ## [](#set-up-monitoring-alerts)Set up monitoring alerts Configure alerts based on these patterns: ### [](#startup-tracking)Startup tracking Alert if `TYPE_INITIALIZING` is not followed by `TYPE_CONNECTION_HEALTHY` within 60 seconds. This indicates the pipeline failed to establish connections during startup. Example alert condition: ```text last_event_type = "TYPE_INITIALIZING" AND time_since_last_event > 60 seconds AND no TYPE_CONNECTION_HEALTHY received ``` ### [](#health-monitoring)Health monitoring Alert if no `TYPE_CONNECTION_HEALTHY` received for a `pipeline_id` for more than 60 seconds. Since heartbeats occur every 30 seconds, missing two consecutive heartbeats indicates the pipeline may have crashed without sending `TYPE_EXITING`. Example alert condition: ```text last_event_type = "TYPE_CONNECTION_HEALTHY" AND time_since_last_event > 60 seconds ``` ### [](#error-detection)Error detection Alert on any `TYPE_CONNECTION_ERROR` events. These indicate connection failures to inputs, outputs, or other external systems. Use the event fields to identify failures: - `connection_errors[].path` - The configuration path of the failing connector (e.g., `input.kafka_franz`) - `connection_errors[].label` - Custom label if assigned to the connector - `connection_errors[].message` - Error message describing the failure Example alert condition: ```text event_type = "TYPE_CONNECTION_ERROR" ``` ### [](#shutdown-tracking)Shutdown tracking Alert on `TYPE_EXITING` events that include an `exit_error.message` field. This indicates unexpected shutdown due to an error rather than graceful termination. Example alert condition: ```text event_type = "TYPE_EXITING" AND exit_error.message IS NOT NULL ``` ## [](#example-monitoring-query)Example monitoring query For a log aggregation system (Splunk, Elasticsearch, etc.): ```text source="__redpanda.connect.status" | where type="TYPE_CONNECTION_ERROR" OR (type="TYPE_EXITING" AND exit_error.message IS NOT NULL) | stats count by pipeline_id, type, connection_errors[0].path ``` This query identifies pipelines with connection errors or unexpected shutdowns. ## [](#troubleshoot-status-events)Troubleshoot status events ### [](#status-events-not-appearing)Status events not appearing If you’ve configured `status_topic` but don’t see events: 1. **Check license**: Status reporting requires a valid Enterprise Edition license - Verify license is installed and valid - Check logs for license validation errors 2. **Verify topic creation**: Ensure the status topic exists and has proper permissions - Create the topic manually if it doesn’t exist: ```bash rpk topic create __redpanda.connect.status --partitions 1 --replicas 3 ``` - Verify Connect instance has write permissions to the topic 3. **Check broker connectivity**: Ensure Connect can reach the Kafka brokers - Verify `seed_brokers` configuration is correct - Check network connectivity and firewall rules 4. **Review configuration**: Verify `status_topic` is set and not empty string ```yaml redpanda: seed_brokers: ["localhost:9092"] pipeline_id: my-pipeline status_topic: __redpanda.connect.status # Must be non-empty ``` ### [](#connection-errors-not-resolving)Connection errors not resolving If you see persistent `TYPE_CONNECTION_ERROR` events: 1. Check the `connection_errors[].path` to identify the failing connector 2. Review configuration for that specific connector (input or output) 3. Verify the external system (database, API, etc.) is accessible 4. Check connector-specific authentication and credentials 5. Review Connect logs for detailed error messages from the connector ### [](#missing-heartbeats)Missing heartbeats If heartbeats (`TYPE_CONNECTION_HEALTHY`) stop but no `TYPE_EXITING` was sent: 1. The pipeline process likely crashed or was killed (SIGKILL) 2. Check system logs for out-of-memory errors or other system issues 3. Check container orchestrator logs (Kubernetes pod events, etc.) 4. Review resource limits and adjust if necessary ## [](#suggested-reading)Suggested reading - [`status_topic` configuration reference](https://docs.redpanda.com/connect/components/redpanda/about/#status_topic) - [Monitoring with metrics and health checks](https://docs.redpanda.com/connect/guides/monitoring/) - [Logger configuration](https://docs.redpanda.com/connect/components/logger/about/) --- # Page 396: Monitoring **URL**: https://docs.redpanda.com/connect/guides/monitoring.md --- # Monitoring --- title: Monitoring latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: monitoring page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: monitoring.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/monitoring.adoc description: Learn how to monitor Redpanda Connect, including setting up metrics and alerts to ensure system health and performance. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-14" --- ## [](#health-checks)Health checks Redpanda Connect serves two HTTP endpoints for health checks: - `/ping` can be used as a liveness probe as it always returns a 200. - `/ready` can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned. ## [](#metrics)Metrics Redpanda Connect [exposes lots of metrics](https://docs.redpanda.com/connect/components/metrics/about/#metric-names) either to Statsd, Prometheus, Cloudwatch or for debugging purposes an HTTP endpoint that returns a JSON formatted object. The target destination of Redpanda Connect metrics is configurable from the [metrics section](https://docs.redpanda.com/connect/components/metrics/about/), where it’s also possible to rename and restrict the metrics that are emitted with mappings. ## [](#pipeline-status-events)Pipeline status events > 📝 **NOTE** > > This component requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. Redpanda Connect can emit real-time status events to a Kafka topic as pipelines start, run, and stop. Status events provide: - Pipeline lifecycle tracking (initialization, running, shutdown) - Connection health monitoring with 30-second heartbeats - Detailed error information when connections fail - Graceful vs. error shutdown detection For configuration and monitoring details, see [Monitor Pipeline Status](https://docs.redpanda.com/connect/guides/monitor-pipeline-status/). ## [](#tracing)Tracing Redpanda Connect also [emits opentracing events](https://docs.redpanda.com/connect/components/tracers/about/) to a tracer of your choice, which can be used to visualise the processors within a pipeline. --- # Page 397: Performance Tuning **URL**: https://docs.redpanda.com/connect/guides/performance_tuning.md --- # Performance Tuning --- title: Performance Tuning latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: performance_tuning page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: performance_tuning.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/performance_tuning.adoc description: Discover best practices for tuning Redpanda Connect performance, optimizing configurations, and achieving maximum throughput. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-10-10" --- ## [](#maximising-io-throughput)Maximising IO throughput This section outlines a few common throughput issues and ways in which they can be solved within Redpanda Connect. It is assumed here that your Redpanda Connect instance is performing only minor processing steps, and therefore has minimal reliance on your CPU resource. If this is not the case the following still applies to an extent, but you should also refer to [the next section regarding CPU utilization](#maximising-cpu-utilization). Firstly, before venturing into Redpanda Connect configurations, you should take an in-depth look at your sources and sinks. Redpanda Connect is generally much simpler architecturally than the inputs and outputs it supports. Spend some time understanding how to squeeze the most out of these services and it will make it easier (or unnecessary) to tune your Redpanda Connect configuration. ### [](#redpanda-connect-reads-too-slowly)Redpanda Connect reads too slowly If Redpanda Connect isn’t reading fast enough from your source it might not necessarily be due to a slow consumer. If the sink is slow this can cause back pressure that throttles the amount Redpanda Connect can read. Try consuming a test feed with the output replaced with `drop`. If you notice that the input consumption suddenly speeds up then the issue is likely with the output, in which case [try the next section](#redpanda-connect-writes-too-slowly). If the `drop` output pipe didn’t help then take a quick look at the basic configuration fields for the input source type. Sometimes there are fields for setting a number of background prefetches or similar concepts that can increase your throughput. For example, increasing the value of `prefetch_count` for an AMQP consumer can greatly increase the rate at which it is consumed. Next, if your source supports multiple parallel consumers then you can try doing that within Redpanda Connect by using a [broker](https://docs.redpanda.com/connect/components/inputs/broker/). For example, if you started with: ```yaml input: http_client: url: http://localhost:4195/get verb: GET ``` You could change to: ```yaml input: broker: copies: 4 inputs: - http_client: url: http://localhost:4195/get verb: GET ``` Which would create the exact same consumer as before with four connections in total. Try increasing the number of copies to see how that affects the throughput. If your multiple consumers would require different configurations then set copies to `1` and write each consumer as a separate object in the `inputs` array. Read the [broker documentation](https://docs.redpanda.com/connect/components/inputs/broker/) for more tips on simplifying broker configs. If your source doesn’t support multiple parallel consumers then unfortunately your options are more limited. A logical next step might be to look at your network/disk configuration to see if that’s a potential cause of contention. ### [](#redpanda-connect-writes-too-slowly)Redpanda Connect Writes Too Slowly If you have an output sink that regularly places back pressure on your source there are a few solutions depending on the details of the issue. Firstly, you should check the config parameters of your output sink. There are often fields specifically for controlling the level of acknowledgement to expect before moving onto the next message, if these levels of guarantee are overkill you can disable them for greater throughput. For example, setting the `ack_replicas` field to `false` in the Kafka sink can have a high impact on throughput. If the config parameters for an output sink aren’t enough then you can try the following: #### [](#increase-in-flight-messages)Increase in flight messages Most outputs have a field `max_in_flight` that allows you to specify how many messages can be in flight at the same time. Increasing this value can improve throughput significantly. #### [](#send-messages-in-batches)Send messages in batches Most outputs will send data quicker when messages are batched, this is often done automatically in the background. However, for a few outputs your batches need to be configured. Read the [batching documentation](https://docs.redpanda.com/connect/configuration/batching/) for more guidance on how to tune message batches within Redpanda Connect. #### [](#level-out-input-spikes-with-a-buffer)Level out input spikes with a buffer There are many reasons why an input source might have spikes or inconsistent throughput rates. It is possible that your output is capable of keeping up with the long term average flow of data, but fails to keep up when an intermittent spike occurs. In situations like these it is sometimes a better use of your hardware and resources to level out the flow of data rather than try and match the peak throughput. This would depend on the frequency and duration of the spikes as well as your latency requirements, and is therefore a matter of judgement. Leveling out the flow of data can be done within Redpanda Connect using a [buffer](https://docs.redpanda.com/connect/components/buffers/about/). Buffers allow an input source to store a bounded amount of data temporarily, which a consumer can work through at its own pace. Buffers always have a fixed capacity, which when full will proceed to block the input just like a busy output would. Therefore, it’s still important to have an output that can keep up with the flow of data, the difference that a buffer makes is that the output only needs to keep up with the _average_ flow of data versus the instantaneous flow of data. For example, if your input usually produces 10 msgs/s, but occasionally spikes to 100 msgs/s, and your output can handle up to 50 msgs/s, it might be possible to configure a buffer large enough to store spikes in their entirety. As long as the average flow of messages from the input remains below 50 msgs/s then your service should be able to continue indefinitely without ever blocking the input source. > 💡 **TIP** > > For a worked example that combines in-flight messages, batching, and buffers, see [Tune the Iceberg output for throughput](https://docs.redpanda.com/connect/guides/iceberg-output-tuning/). ## [](#maximising-cpu-utilization)Maximising CPU utilization Some [processors](https://docs.redpanda.com/connect/components/processors/about/) within Redpanda Connect are relatively heavy on your CPU, and can potentially become the bottleneck of a service. In these circumstances it is worth configuring Redpanda Connect so that your processors are running on each available core of your machine without contention. An array of processors in any section of a Redpanda Connect config becomes a single logical pipeline of steps running on a single logical thread. The easiest way to create parallel processor threads is to configure them inside the [pipeline](https://docs.redpanda.com/connect/configuration/processing_pipelines/) configuration block, where we can explicitly set any number of parallel processor threads independent of how many inputs or outputs we want to use. Please refer [to the documentation regarding pipelines](https://docs.redpanda.com/connect/configuration/processing_pipelines/) for some examples. --- # Page 398: Serverless **URL**: https://docs.redpanda.com/connect/guides/serverless/about.md --- # Serverless --- title: Serverless latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: serverless/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: serverless/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/serverless/about.adoc description: Learn about serverless architecture in Redpanda Connect, including benefits, use cases, and implementation details. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Currently the only supported serverless deployment of Redpanda Connect is for [AWS Lambda](https://docs.redpanda.com/connect/guides/serverless/lambda/), if you are interested in other platforms please [raise an issue](https://github.com/redpanda-data/connect/issues). ## [](#platforms)Platforms - [AWS Lambda](https://docs.redpanda.com/connect/guides/serverless/lambda/) --- # Page 399: Lambda **URL**: https://docs.redpanda.com/connect/guides/serverless/lambda.md --- # Lambda --- title: Lambda latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: serverless/lambda page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: serverless/lambda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/serverless/lambda.adoc description: Deploying as an AWS Lambda function page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- You can use either of the following distributions to deploy Redpanda Connect as an AWS Lambda function: - The `redpanda-connect-lambda` distribution is specifically tailored for deployment as an AWS Lambda function on the `go1.x` runtime, which runs Amazon Linux on the `x86_64` architecture. - The `redpanda-connect-lambda-al2` distribution supports the `provided.al2` runtime, which runs Amazon Linux 2 on either the `x86_64` or `arm64` architecture. ## [](#configuration)Configuration The AWS Lambda version of Redpanda Connect uses the same configuration format as a regular instance. Choose from the following two options: 1. Inline via the `BENTHOS_CONFIG` environment variable (YAML format). 2. Via the filesystem using a layer, extension, or container image. By default, the `redpanda-connect-lambda` distribution looks for a valid configuration file in the locations listed below. Alternatively, you can set the configuration file path explicity by passing a `BENTHOS_CONFIG_PATH` environment variable. - `./benthos.yaml` - `./config.yaml` - `/benthos.yaml` - `/etc/benthos/config.yaml` - `/etc/benthos.yaml` Also, the `http`, `input` and `buffer` sections are ignored as the service wide HTTP server is not used, and messages are inserted via function invocations. If the `output` section is omitted in your config then the result of the processing pipeline is returned back to the caller, otherwise the resulting data is sent to the output destination. ## [](#run-with-an-output)Run with an output The flow of a Redpanda Connect Lambda function with an output configured looks like this: ```text redpanda-connect-lambda +------------------------------+ | | -------> Processors ----> Output -----> Somewhere invoke | | | <-------------------------------------------/ | | | | +------------------------------+ ``` Where the call will block until the output target has confirmed receipt of the resulting payload. When the message is successfully propagated a JSON payload is returned of the form `{"message":"request successful"}`, otherwise an error is returned containing the reason for the failure. ## [](#run-without-an-output)Run without an output The flow when an output is not configured looks like this: ```text redpanda-connect-lambda +--------------------+ | | -------> Processors --\ | invoke | | | <---------------------/ | | | | | +--------------------+ ``` Where the function returns the result of processing directly back to the caller. The format of the result differs depending on the number of batches and messages of a batch that resulted from the invocation: - Single message of a single batch: `{}` (JSON object) - Multiple messages of a single batch: `[{},{}]` (Array of JSON objects) - Multiple batches: `[[{},{}],[{}]]` (Array of arrays of JSON objects, batches of size one are a single object array in this case) ### [](#process-errors)Process errors By default, the Redpanda Connect Lambda handler fails if messages encounter an uncaught error during execution. ## [](#run-a-combination)Run a combination It’s possible to configure pipelines that send messages to third party destinations and also return a result back to the caller. This is done by configuring an output block and including an output of the type `sync_response`. For example, if we wished for our Lambda function to send a payload to Kafka and also return the same payload back to the caller we could use a [broker](https://docs.redpanda.com/connect/components/outputs/broker/): ```yml output: broker: pattern: fan_out outputs: - kafka: addresses: - todo:9092 client_id: benthos_serverless topic: example_topic - sync_response: {} ``` ## [](#upload-to-aws)Upload to AWS ### [](#go1-x-on-x86_64)go1.x on x86_64 Grab an archive labelled `redpanda-connect-lambda` from the [releases page](https://github.com/redpanda-data/connect/releases) page and then create your function: ```sh LAMBDA_ENV=`cat yourconfig.yaml | jq -csR {Variables:{BENTHOS_CONFIG:.}}` aws lambda create-function \ --runtime go1.x \ --handler redpanda-connect-lambda \ --role benthos-example-role \ --zip-file fileb://redpanda-connect-lambda_4.113.1_linux_amd64.zip \ --environment "$LAMBDA_ENV" \ --function-name benthos-example ``` ### [](#provided-al2-on-arm64)provided.al2 on arm64 Grab an archive labelled `redpanda-connect-lambda-al2` for `arm64` from the [releases page](https://github.com/redpanda-data/connect/releases) page and then create your function (AWS CLI v2 only): ```sh LAMBDA_ENV=`cat yourconfig.yaml | jq -csR {Variables:{BENTHOS_CONFIG:.}}` aws lambda create-function \ --runtime provided.al2 \ --architectures arm64 \ --handler not.used.for.provided.al2.runtime \ --role benthos-example-role \ --zip-file fileb://redpanda-connect-lambda-al2_4.113.1_linux_arm64.zip \ --environment "$LAMBDA_ENV" \ --function-name benthos-example ``` Note that you can also run `redpanda-connect-lambda-al2` on `x86_64`, just use the `amd64` zip instead. ## [](#invoke)Invoke ```sh aws lambda invoke \ --function-name benthos-example \ --payload '{"your":"document"}' \ out.txt && cat out.txt && rm out.txt ``` ## [](#build)Build You can build and archive the function yourself with: ```sh go build github.com/redpanda-data/connect/v4/cmd/serverless/connect-lambda zip connect-lambda.zip connect-lambda ``` --- # Page 400: Streams Mode **URL**: https://docs.redpanda.com/connect/guides/streams_mode/about.md --- # Streams Mode --- title: Streams Mode latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: streams_mode/about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: streams_mode/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/streams_mode/about.adoc description: Get an overview of streams mode in Redpanda Connect, detailing its features, use cases, and setup instructions. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- A Redpanda Connect stream consists of four components; an input, an optional buffer, processor pipelines and an output. Under normal use a Redpanda Connect instance is a single stream, and these components are configured within the service config file. Alternatively, Redpanda Connect can be run in `streams` mode, where a single running Redpanda Connect instance is able to run multiple entirely isolated streams. Adding streams in this mode can be done in two ways: 1. [Static configuration files](https://docs.redpanda.com/connect/guides/streams_mode/using_config_files/) allows you to maintain a directory of static stream configuration files that will be traversed by Redpanda Connect. 2. An [HTTP REST API](https://docs.redpanda.com/connect/guides/streams_mode/using_rest_api/) allows you to dynamically create, read the status of, update, and delete streams at runtime. These two methods can be used in combination, i.e. it’s possible to update and delete streams that were created with static files. When running Redpanda Connect in streams mode it is still necessary to provide a general service wide configuration with the `-o`/`--observability` flag that specifies observability configuration such as the `metrics`, `logger` and `tracing` sections, as well the `http` section for configuring how the HTTP server should behave. You can import resources either in the general configuration, or using the `-r`/`--resources` flag, the same as when running Redpanda Connect in regular mode. ```bash rpk connect streams -r "./prod/*.yaml" -o ./config.yaml ``` ## [](#http-endpoints)HTTP endpoints A Redpanda Connect config can contain components such as an `http_server` input that register endpoints to the service-wide HTTP server. When these components are created from within a named stream in streams mode the endpoint will be prefixed with the streams identifier by default. For example, a stream with the identifier `foo` and the config: ```yaml input: http_server: path: /meow pipeline: processors: - mapping: 'root = "meow " + content()' output: sync_response: {} ``` Will register an endpoint `/meow`, which will be prefixed with the name `foo` to become `/foo/meow`. This behavior is intended to make a clearer distinction between endpoints registered by different streams, and prevent collisions of those endpoints. However, you can disable this behavior by setting the flag `--prefix-stream-endpoints` to `false` (`rpk connect streams --prefix-stream-endpoints=false ./streams/*.yaml`). ## [](#resources)Resources When running Redpanda Connect in streams mode [resource components](https://docs.redpanda.com/connect/configuration/resources/) are shared across all streams. The streams mode HTTP API also provides an endpoint for modifying and adding resource configurations dynamically. ## [](#metrics)Metrics Metrics from all streams are aggregated and exposed via the method specified in [the config](https://docs.redpanda.com/connect/components/metrics/about/) of the Redpanda Connect instance running in `streams` mode, with their metrics enriched with the tag `stream` containing the stream name. For example, a Redpanda Connect instance running in streams mode running a stream named `foo` would have metrics from `foo` registered with the label `stream` with the value of `foo`. This can cause problems if your streams are short lived and uniquely named as the number of metrics registered will continue to climb indefinitely. In order to avoid this you can use the `mapping` field to filter metric names. ```yaml # Only register metrics for the stream `foo`. Others will be ignored. metrics: mapping: if meta("stream") != "foo" { deleted() } prometheus: {} ``` --- # Page 401: Streams API **URL**: https://docs.redpanda.com/connect/guides/streams_mode/streams_api.md --- # Streams API --- title: Streams API latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: streams_mode/streams_api page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: streams_mode/streams_api.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/streams_mode/streams_api.adoc description: Get an overview of streams mode in Redpanda Connect, detailing its features, use cases, and setup instructions. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- When Redpanda Connect is run in `streams` mode it will open up an HTTP REST API for creating and managing independent streams of data instead of creating a single stream. Each stream has its own input, buffer, pipeline and output sections which contains an isolated stream of data with its own lifetime. A stream config cannot include [resources](https://docs.redpanda.com/connect/configuration/resources/), and instead these should be created and modified using the `/resources/{type}/{id}` endpoint. A walkthrough on using this API [can be found here](https://docs.redpanda.com/connect/guides/streams_mode/using_rest_api/). ## [](#api)API ### [](#get-ready)GET `/ready` Returns a 200 OK response if all active streams are connected to their respective inputs and outputs at the time of the request. Otherwise, a 503 response is returned along with a message naming the faulty stream. If zero streams are active this endpoint still returns a 200 OK response. ### [](#get-streams)GET `/streams` Returns a map of existing streams by their unique identifiers to an object showing their status and uptime. #### [](#response-200)Response 200 ```json { "": { "active": "", "uptime": "", "uptime_str": "" } } ``` ### [](#post-streams)POST `/streams` Sets the entire collection of streams to the body of the request. Streams that exist but aren’t within the request body are _removed_, streams that exist already and are in the request body are updated, other streams within the request body are created. ```json { "": "" } ``` #### [](#response-200-2)Response 200 The streams were updated successfully. #### [](#response-400)Response 400 A configuration was invalid, or has linting errors. If linting errors were detected then a JSON response is provided of the form: ```json { "linting_errors": [ "", "uptime": "", "uptime_str": "", "config": "" } ``` ### [](#put-streamsid)PUT `/streams/{id}` Update an existing stream identified by `id` by posting a body containing the new stream configuration in either JSON or YAML format. The configuration should be a standard Redpanda Connect configuration containing the sections `input`, `buffer`, `pipeline` and `output`. The previous stream will be shut down before and a new stream will take its place. #### [](#response-200-5)Response 200 The stream was updated successfully. #### [](#response-400-3)Response 400 The configuration was invalid, or has linting errors. If linting errors were detected then a JSON response is provided of the form: ```json { "linting_errors": [ " ./streams/foo.yaml < ./streams/bar.yaml < Input (AMQP) -> Processors -> Output (AMQP) <------- Acknowledgement --------- ``` However, Redpanda Connect supports bidirectional protocols like HTTP and WebSocket, which allow responses to be returned directly from the pipeline. For example, HTTP is a request/response protocol, and inputs like `http_server` (Self-Managed) or `gateway` (Redpanda Cloud) support returning response payloads to the requester. ```text --------- Request Body --------> Input (HTTP) -> Processors -> Output (Sync Response) <--- Response Body (and ack) --- ``` ## [](#routing-processed-messages-back)Routing processed messages back To return a processed response, use the [`sync_response`](https://docs.redpanda.com/connect/components/outputs/sync_response/) output. Use the `http_server` input in Self-Managed deployments: ```yaml input: http_server: path: /weather pipeline: processors: - mapping: | root = { city: json("location"), forecast: "Clear skies with light winds", temperature_c: 22 } output: sync_response: {} ``` Sending this request: ```json { "location": "Berlin" } ``` Returns: ```json { "city": "Berlin", "forecast": "Clear skies with light winds", "temperature_c": 22 } ``` ## [](#combine-with-other-outputs)Combine with other outputs You can route processed messages to storage and return a response using a [`broker`](https://docs.redpanda.com/connect/components/outputs/broker/) output. ```yaml input: http_server: path: /weather output: broker: pattern: fan_out outputs: - kafka: addresses: [localhost:9092] topic: weather.requests - sync_response: processors: - mapping: | root = { status: "received", received_at: now() } ``` ## [](#returning-partially-processed-messages)Returning partially processed messages You can return a response before the message is fully processed by using the [`sync_response` processor](https://docs.redpanda.com/connect/components/processors/sync_response/). This allows continued processing after the response is set. ```yaml pipeline: processors: - mapping: root = "Received weather report for %s".format(json("location")) - sync_response: {} - mapping: root.reported_at = now() ``` This returns `"Received weather report for Berlin"` to the client, but continues modifying the message before storing or forwarding it. > 📝 **NOTE** > > Due to delivery guarantees, the response is not sent until all downstream processing and acknowledgements are complete. ## [](#routing-output-responses-back)Routing output responses back Some outputs, such as [`http_client`](https://docs.redpanda.com/connect/components/outputs/http_client/), support returning a downstream response back to the input. ```yaml input: http_server: path: /proxy-weather output: http_client: url: https://api.example.com/forecast verb: POST propagate_response: true ``` This forwards the request to an external weather API and returns its response to the original requester. You can combine this with other outputs: ```yaml input: http_server: path: /proxy-weather output: broker: pattern: fan_out outputs: - kafka: addresses: [localhost:9092] topic: weather.proxy - http_client: url: https://api.example.com/forecast verb: POST propagate_response: true ``` --- # Page 405: Install or Upgrade Redpanda Connect **URL**: https://docs.redpanda.com/connect/install.md --- # Install or Upgrade Redpanda Connect --- title: Install or Upgrade Redpanda Connect latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/index.adoc description: Discover all ways to install or upgrade Redpanda Connect. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-10-10" --- This page helps you choose how to install or upgrade Redpanda Connect. Select the method that best fits your needs: | Method | When to use | Limitations | | --- | --- | --- | | Redpanda CLI (recommended) | The simplest way to install, upgrade, and manage Redpanda Connect. Works on all supported platforms. Automatically upgrades with a single command. | Doesn’t include the connectors that need cgo. For the list, see Install or Upgrade from a Prebuilt Binary. | | Helm chart (Kubernetes) | Deploy and manage Redpanda Connect on Kubernetes clusters using Helm. Supports upgrades and rollbacks. | Requires a running Kubernetes cluster and Helm. Not for bare metal or VM installs. | | Redpanda Operator (Kubernetes, beta) | Run pipelines as declaratively managed Pipeline resources on Kubernetes. Best when you already manage Redpanda with the Redpanda Operator. You don’t install a separate operator. | Beta feature. Requires an Enterprise Edition license and Redpanda Operator 26.2 or later. | | Prebuilt binary (advanced users) | Use when you want to manage the binary yourself, or need the cgo-enabled binary for the connectors that need cgo. Standard archives are published for Linux, macOS, and Windows. | The cgo-enabled archive is available for Linux AMD64 only. You must manage upgrades manually. | | Build from source (power users / contributors) | Gives full control. Lets you build with CGO_ENABLED=1 and the x_benthos_extra build tag to include the connectors that need cgo. | Requires the Go toolchain. The cgo-enabled build also needs a C compiler and the ZeroMQ library (libzmq). | --- # Page 406: Build or Upgrade from Source **URL**: https://docs.redpanda.com/connect/install/build-from-source.md --- # Build or Upgrade from Source --- title: Build or Upgrade from Source latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: build-from-source page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: build-from-source.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/build-from-source.adoc description: Build or upgrade Redpanda Connect from source. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-10-10" --- You can build Redpanda Connect in two ways. Each build command on this page matches the Redpanda Connect release build of the same type. - **Standard build**: Suitable for most users. Builds with cgo disabled, like the standard release binaries, and leaves out the components that need cgo. - **cgo-enabled build**: Required only for the components listed below. The build needs a C compiler and the ZeroMQ development headers (`libzmq`). These components are available only in the cgo-enabled build: - [`tigerbeetle_cdc` input](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/) - [`zmq4` input](https://docs.redpanda.com/connect/components/inputs/zmq4/) - [`zmq4` output](https://docs.redpanda.com/connect/components/outputs/zmq4/) - [`ffi` processor](https://docs.redpanda.com/connect/components/processors/ffi/) Each component’s reference page also notes when the component needs cgo. ## [](#install-from-source)Install from source 1. Ensure you have Go installed on your system. You can download it from [the official Go website](https://golang.org/dl/). 2. For the cgo-enabled build only, install a C compiler and the ZeroMQ library with its development headers. For example, on Debian or Ubuntu: ```bash sudo apt-get install build-essential libzmq3-dev ``` The machine that runs a cgo-enabled build also needs the ZeroMQ shared library, because the binary links to it dynamically. 3. Clone the Redpanda Connect repository: ```bash git clone https://github.com/redpanda-data/connect.git cd connect ``` 4. Build the binary (choose one): | Build type | Build command | | --- | --- | | Standard (no cgo, most users) | CGO_ENABLED=0 go build -tags timetzdata -o redpanda-connect ./cmd/redpanda-connect | | cgo-enabled (adds the components that need cgo) | CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata -o redpanda-connect ./cmd/redpanda-connect | The `timetzdata` build tag embeds the time zone database in the binary, so time zone lookups work on systems that don’t have one installed. The `x_benthos_extra` build tag adds the components that link to external libraries. 5. Move the binary to a directory in your `PATH` (for example, `/usr/local/bin`): ```bash sudo mv redpanda-connect /usr/local/bin/ ``` 6. Verify the installation: ```bash redpanda-connect --version ``` You should see a version. ## [](#upgrade-a-source-build)Upgrade a source build 1. Navigate to your existing clone of the repository: ```bash cd connect ``` 2. Fetch the latest tags and check out the version you want: ```bash git fetch --all git checkout v4.113.1 ``` 3. Rebuild the binary (choose one): | Build type | Build command | | --- | --- | | Standard (no cgo, most users) | CGO_ENABLED=0 go build -tags timetzdata -o redpanda-connect ./cmd/redpanda-connect | | cgo-enabled (adds the components that need cgo) | CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata -o redpanda-connect ./cmd/redpanda-connect | 4. Replace the previous binary and verify: ```bash sudo mv redpanda-connect /usr/local/bin/ redpanda-connect --version ``` > 📝 **NOTE** > > After upgrading (or downgrading), you must restart a pipeline to use features in the Redpanda Connect version you have deployed. ## [](#uninstall)Uninstall To uninstall Redpanda Connect, remove the installed binary: ```sh sudo rm /usr/local/bin/redpanda-connect ``` This removes the Redpanda Connect binary from your system. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](https://docs.redpanda.com/connect/components/about/). --- # Page 407: Install or Upgrade with the Helm Chart **URL**: https://docs.redpanda.com/connect/install/helm-chart.md --- # Install or Upgrade with the Helm Chart --- title: Install or Upgrade with the Helm Chart latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: helm-chart page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: helm-chart.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/helm-chart.adoc description: Deploy or upgrade Redpanda Connect on Kubernetes using Helm. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- The official Redpanda Connect Helm chart lets you deploy and manage Redpanda Connect on Kubernetes. ## [](#install-with-helm)Install with Helm To deploy Redpanda Connect on Kubernetes, use the official Helm chart: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda-connect redpanda/connect --namespace --create-namespace ``` Replace `` with your desired Kubernetes namespace. > 💡 **TIP** > > See [Get Started with the Redpanda Connect Helm Chart](https://docs.redpanda.com/connect/get-started/quickstarts/helm-chart/) for a full walkthrough and configuration options. ## [](#upgrade-with-helm)Upgrade with Helm To upgrade Redpanda Connect to a newer Helm chart version: 1. Ensure your Helm repository is updated to fetch the latest version of the Redpanda Helm chart: ```bash helm repo update ``` 2. Upgrade your existing deployment using the [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/) command: ```bash helm upgrade redpanda-connect redpanda/connect --namespace --values ``` Replace `` with the path to your custom configuration file. ### [](#roll-back-an-upgrade)Roll back an upgrade If an upgrade introduces issues, you can roll back to a previous release. Helm retains previous deployments, allowing you to roll back with the [`helm rollback`](https://helm.sh/docs/helm/helm_rollback/) command: ```bash helm rollback redpanda-connect --namespace ``` To find the revision number, use the [`helm history`](https://helm.sh/docs/helm/helm_history/) command: ```bash helm history redpanda-connect --namespace ``` This command lists previous deployments and their revision numbers. ## [](#uninstall)Uninstall To uninstall Redpanda Connect from your Kubernetes cluster: ```bash helm uninstall redpanda-connect --namespace ``` Replace `` with the namespace where Redpanda Connect was installed. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](https://docs.redpanda.com/connect/components/about/). --- # Page 408: Run Pipelines with the Redpanda Operator **URL**: https://docs.redpanda.com/connect/install/kubernetes-operator.md --- # Run Pipelines with the Redpanda Operator --- title: Run Pipelines with the Redpanda Operator page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: kubernetes-operator page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: kubernetes-operator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/kubernetes-operator.adoc description: Run Redpanda Connect pipelines declaratively on Kubernetes with the Redpanda Operator's Pipeline resource. # Beta release status page-beta: "true" page-git-created-date: "2026-07-28" page-git-modified-date: "2026-07-28" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Run Redpanda Connect pipelines as declaratively managed Kubernetes workloads. You define each pipeline in a `Pipeline` custom resource, and the Redpanda Operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. > 📝 **NOTE** > > This feature is in beta and requires an Enterprise Edition license and Redpanda Operator 26.2 or later. Beta features are not recommended for production environments. To give feedback, reach out to the Redpanda team in [Redpanda Community Slack](https://redpanda.com/slack). You don’t install a separate operator. The Connect controller is part of the Redpanda Operator, and you enable it with a single Helm setting (`connectController.enabled=true`) on your existing installation. With the Pipeline resource, the operator: - Injects broker addresses, TLS material, and SASL credentials for a referenced Redpanda cluster into the rendered configuration. - Projects secrets and other values into the pipeline Pod as environment variables for `${NAME}` interpolation. - Implements the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or KEDA can drive the replica count, including on consumer group lag. - Exposes Prometheus metrics and, with monitoring enabled, renders a PodMonitor per pipeline. The complete guides live in the Redpanda Self-Managed documentation: - [Run Redpanda Connect Pipelines in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-connect-pipelines/): enable the Connect controller, deploy pipelines, connect them to clusters, target external clusters, inject secrets and cloud IAM identities, customize the workload, monitor, and troubleshoot. - [Autoscale Redpanda Connect Pipelines](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-autoscale-connect-pipelines/): step-by-step HPA and KEDA autoscaling. - [Pipeline CRD reference](https://docs.redpanda.com/streaming/current/reference/k-crd/#k8s-api-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-pipeline): all Pipeline fields. To run Redpanda Connect on Kubernetes without the operator, see [Install or Upgrade with the Helm Chart](https://docs.redpanda.com/connect/install/helm-chart/). --- # Page 409: Install or Upgrade from a Prebuilt Binary **URL**: https://docs.redpanda.com/connect/install/prebuilt-binary.md --- # Install or Upgrade from a Prebuilt Binary --- title: Install or Upgrade from a Prebuilt Binary latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: prebuilt-binary page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: prebuilt-binary.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/prebuilt-binary.adoc description: Download, install, or upgrade Redpanda Connect from a prebuilt binary. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-10-10" --- You can choose between two types of prebuilt binaries: - **Standard binary**: Suitable for most users. Built with cgo disabled, and available for Linux, macOS, and Windows on AMD64 and ARM64. Leaves out the components that need cgo. - **cgo-enabled binary**: Required only for the components listed below. Available for Linux AMD64 only. The machine that runs it needs the ZeroMQ shared library (`libzmq`), because the binary links to it dynamically. These components are available only in the cgo-enabled binary: - [`tigerbeetle_cdc` input](https://docs.redpanda.com/connect/components/inputs/tigerbeetle_cdc/) - [`zmq4` input](https://docs.redpanda.com/connect/components/inputs/zmq4/) - [`zmq4` output](https://docs.redpanda.com/connect/components/outputs/zmq4/) - [`ffi` processor](https://docs.redpanda.com/connect/components/processors/ffi/) Each component’s reference page also notes when the component needs cgo. ## [](#install-a-prebuilt-binary)Install a prebuilt binary 1. Ensure you have `wget` and `tar` installed. 2. Download the desired Redpanda Connect archive for your platform. The examples download the Linux AMD64 archives. For other platforms, replace `linux_amd64` with `darwin_amd64`, `darwin_arm64`, `linux_arm64`, `windows_amd64`, or `windows_arm64`. You can find a complete list of releases on the [GitHub releases page](https://github.com/redpanda-data/connect/releases). | Binary type | Example download command | | --- | --- | | Standard (no cgo, most users) | wget https://github.com/redpanda-data/connect/releases/download/v4.113.1/redpanda-connect_4.113.1_linux_amd64.tar.gz | | cgo-enabled (adds the components that need cgo) | wget https://github.com/redpanda-data/connect/releases/download/v4.113.1/redpanda-connect-cgo_4.113.1_linux_amd64.tar.gz | 3. Extract the binary from the archive (replace the filename as needed): ```bash tar -xzf .tar.gz ``` 4. Move the binary to a directory in your `PATH` (for example, `/usr/local/bin`): ```bash sudo mv redpanda-connect /usr/local/bin/ ``` 5. Verify the installation: ```bash redpanda-connect --version ``` You should see a version number. ## [](#upgrade-a-prebuilt-binary)Upgrade a prebuilt binary To upgrade, repeat the installation steps with the new version. Stop any running Redpanda Connect processes, download and extract the new binary, replace the old binary, and verify the upgrade. > 📝 **NOTE** > > After upgrading (or downgrading), you must restart a pipeline to use features in the Redpanda Connect version you have deployed. ## [](#uninstall)Uninstall To uninstall Redpanda Connect, remove the installed binary: ```sh sudo rm /usr/local/bin/redpanda-connect ``` This removes the Redpanda Connect binary from your system. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](https://docs.redpanda.com/connect/components/about/). --- # Page 410: Install or Upgrade with rpk **URL**: https://docs.redpanda.com/connect/install/rpk.md --- # Install or Upgrade with rpk --- title: Install or Upgrade with rpk latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: rpk page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rpk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/rpk.adoc description: Install or upgrade Redpanda Connect using the Redpanda CLI. page-git-created-date: "2025-09-25" page-git-modified-date: "2026-08-12" --- The Redpanda CLI (`rpk`) is the recommended way to install, upgrade, and manage Redpanda Connect on bare metal. It provides a simple, automated workflow for most users. ## [](#install)Install To install Redpanda Connect using the Redpanda CLI, choose your platform: ### [](#linux)Linux To install, or update to, the latest version of `rpk` for Linux, run: #### amd64 ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-linux-amd64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && sha256sum --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-linux-arm64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && sha256sum --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. To install, or update to, a version other than the latest, run the following command. It verifies the archive against the published SHA-256 checksums before unzipping it: #### amd64 ```bash curl -fLO https://rpk.redpanda.com/v/rpk-linux-amd64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && sha256sum --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -fLO https://rpk.redpanda.com/v/rpk-linux-arm64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && sha256sum --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` #### [](#fips-compliance)FIPS compliance > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. To install `rpk` to run the latest version of Redpanda Connect in FIPS-compliant mode, you must install the `redpanda-rpk-fips` and `redpanda-connect-fips` packages. Both packages are built using the [Microsoft GoLang compiler](https://github.com/microsoft/go) and the [Microsoft’s Go Crypto OpenSSL package](https://github.com/microsoft/go-crypto-openssl), which uses the FIPS-approved version of OpenSSL. > ❗ **IMPORTANT** > > The packages for FIPS-compliant `rpk` (`redpanda-rpk-fips`) and Redpanda `rpk` (`redpanda-rpk`) are mutually exclusive, and so cannot be installed in the same environment. ##### RHEL 1. To make sure your distribution is up to date, run: ```bash sudo dnf upgrade ``` 2. Add `redpanda` to your `dnf` list of repositories. ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo dnf install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with Redpanda Connect releases, run the following command: sudo dnf update ##### Debian/Ubuntu 1. To make sure your distribution is up to date, run: ```bash sudo apt upgrade ``` 2. Add `redpanda` to your `apt` list of repositories. ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo apt install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with the Redpanda Connect releases, run the following command: sudo apt update ### [](#macos)MacOS #### Homebrew 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` #### Manual Download To install or update `rpk` through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, select **Apple Silicon**. ##### Intel macOS To install, or update to, the latest version of `rpk` for Intel macOS, run: ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-darwin-amd64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && shasum -a 256 --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run the following command. It verifies the archive against the published SHA-256 checksums before unzipping it: ```bash curl -fLO https://rpk.redpanda.com/v/rpk-darwin-amd64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && shasum -a 256 --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` ##### Apple Silicon To install, or update to, the latest version of `rpk` for Apple Silicon, run: ```bash curl -fLO https://rpk.redpanda.com/latest/rpk-darwin-arm64.zip && curl -fLO https://rpk.redpanda.com/latest/rpk_checksums.txt && shasum -a 256 --ignore-missing -c rpk_checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run the following command. It verifies the archive against the published SHA-256 checksums before unzipping it: ```bash curl -fLO https://rpk.redpanda.com/v/rpk-darwin-arm64.zip && curl -fLO https://rpk.redpanda.com/v/rpk__checksums.txt && shasum -a 256 --ignore-missing -c rpk__checksums.txt && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` ### [](#docker)Docker To use `rpk` in Docker, you can use the Redpanda Docker image, which includes both Redpanda and `rpk` as part of the installation. ```bash docker pull docker.redpanda.com/redpandadata/connect ``` ## [](#upgrade)Upgrade Redpanda Connect uses the `rpk connect` plugin to manage upgrades, which is automatically installed when you start to run `rpk connect` commands from the `rpk` CLI. You do not need to upgrade the entire Redpanda CLI. The reverse is also true: upgrading `rpk` does not upgrade Redpanda Connect. The plugin is a separate binary that stays at its installed version, even across `rpk` upgrades, until you run `rpk connect upgrade`. The automatic install happens only on first use and installs the latest version at that moment. Check the version of Redpanda Connect already installed: ```bash rpk connect --version ``` ### [](#upgrade-to-the-latest-version)Upgrade to the latest version The simplest way to upgrade is to use the `rpk connect upgrade` command: ```bash rpk connect upgrade ``` Example output: ```bash Redpanda Connect successfully upgraded from 4.38.0 to the latest version (4.113.1). ``` For a full list of `rpk connect upgrade` options, run: ```bash rpk connect upgrade --help ``` ### [](#upgrade-to-a-specific-version)Upgrade to a specific version Force an upgrade (or downgrade) to a specific version of Redpanda Connect using the following command: ```bash rpk connect install --connect-version --force ``` Replace the `` placeholder with the full version number, for example: `4.38.0`. You may need to wait a few seconds before the installation completes. Example output: ```bash Redpanda Connect 4.38.0 successfully installed. ``` ## [](#uninstall)Uninstall If you no longer want to use Redpanda Connect, you can uninstall the `rpk connect` plugin from the CLI: ```bash rpk connect uninstall ``` Example output: ```bash PATH MESSAGE /Users/username/.local/bin/.rpk.managed-connect OK ``` > 📝 **NOTE** > > After upgrading (or downgrading), you must restart a pipeline to use features in the Redpanda Connect version you have deployed. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](https://docs.redpanda.com/connect/components/about/). --- # Page 411: Plugins **URL**: https://docs.redpanda.com/connect/plugins/about.md --- # Plugins --- title: Plugins latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/plugins/pages/about.adoc description: Create and load dynamic plugins at runtime with gRPC. page-git-created-date: "2025-06-11" page-git-modified-date: "2025-06-11" --- Redpanda Connect’s RPC plugin framework is a powerful Apache 2.0 licensed feature that takes data streaming flexibility to the next level. By enabling you to create and load plugins at runtime using any programming language that supports gRPC, this framework opens up a world of new integration possibilities beyond Go. Whether you need to leverage specialized libraries, integrate with AI/ML tools, or build custom components in your preferred language, Redpanda Connect plugins provide the extensibility and interoperability to make it happen. ## [](#plugin-types)Plugin types Redpanda Connect supports two types of plugins: Compiled plugins Built in Go and compiled directly into the Redpanda Connect binary. These offer maximum performance for critical workloads. Dynamic plugins External executables that communicate with Redpanda Connect via gRPC. These provide language flexibility and can be loaded at runtime without rebuilding the binary. This documentation focuses on dynamic plugins. ## [](#when-to-use-dynamic-plugins)When to use dynamic plugins Dynamic plugins are ideal when you need to extend Redpanda Connect’s capabilities without the constraints of Go. Use cases include: - Use existing libraries not available in Go - Write plugins in languages other than Go (such as Python) - Add plugins without rebuilding the entire Redpanda Connect binary - Deploy plugins independently of your main Redpanda Connect deployment - Integrate with AI/ML frameworks, data science libraries, or specialized tools ## [](#architecture)Architecture Dynamic plugins run as separate processes and communicate with the main Redpanda Connect process through gRPC over Unix sockets. This architecture provides the following benefits: - Process isolation: Plugin crashes don’t affect the main engine. - Language agnostic: Support for any language with gRPC libraries. - Modular design: Each plugin maps to a single subprocess. - Battle-tested communication. Uses proven gRPC technology. The system includes three compiled plugin wrappers (one for each component type: `BatchInput`, `BatchProcessor`, and `BatchOutput`) that handle the communication with external plugin executables. ## [](#supported-languages)Supported languages Redpanda Connect currently provides the following SDKs: - **Go**: Familiar environment for existing Redpanda Connect developers with type-safe interfaces - **Python**: Access to rich ecosystem including PyTorch, TensorFlow, Hugging Face Transformers, LangChain, Pandas, and NumPy ## [](#performance-considerations)Performance considerations Dynamic plugins introduce some serialization and IPC overhead compared to compiled plugins. To minimize performance impact: - The system uses batch components exclusively to amortize cross-process communication costs. - Each plugin runs in a separate process with predictable resource usage. - For maximum performance in critical workloads, use compiled plugins instead. ## [](#prerequisites)Prerequisites Before creating dynamic plugins, ensure you have: - Redpanda Connect v4.56.0 or later - Python environment (for Python plugins) - Required language SDKs installed ### [](#install-redpanda-connect)Install Redpanda Connect Verify your version, and upgrade if necessary: ```shell rpk --version rpk connect upgrade ``` ## [](#create-your-first-plugin)Create your first plugin This example demonstrates creating a simple FizzBuzz processor plugin in Python. ### [](#step-1-set-up-the-environment)Step 1: Set up the environment ```shell # Create a new directory for your plugin mkdir fizzbuzz_plugin cd fizzbuzz_plugin # Initialize with link:https://docs.astral.sh/uv/[uv] (or use pip/conda) uv init . uv add redpanda_connect ``` ### [](#step-2-download-the-example)Step 2: Download the example ```shell # Download the example processor curl -o fizzbuzz_processor.py \ https://raw.githubusercontent.com/redpanda-data/connect/main/public/plugin/python/examples/fizzbuzz_processor.py ``` ### [](#step-3-create-the-plugin-configuration)Step 3: Create the plugin configuration Create `plugin.yaml`: ```yaml name: fizzbuzz summary: Transforms numbers according to FizzBuzz rules (3=Fizz, 5=Buzz, 15=FizzBuzz) command: ["uv", "run", "fizzbuzz_processor.py"] type: processor fields: [] ``` ### [](#step-4-create-the-pipeline-configuration)Step 4: Create the pipeline configuration Create `connect.yaml`: ```yaml input: generate: interval: 1s mapping: | root = counter() % 20 + 1 # Generate numbers 1-20 pipeline: processors: - fizzbuzz: {} output: stdout: codec: lines ``` ### [](#step-5-run-the-plugin)Step 5: Run the plugin Execute the pipeline with your dynamic plugin: ```shell rpk connect run --rpc-plugins=plugin.yaml connect.yaml ``` Expected output: ```text 1 2 Fizz 4 Buzz Fizz 7 8 Fizz Buzz 11 Fizz 13 14 FizzBuzz 16 17 Fizz 19 Buzz ``` ## [](#python-plugin-development)Python plugin development ### [](#basic-processor-structure)Basic processor structure Python processors use the `@redpanda_connect.processor` decorator: ```python import asyncio import logging import redpanda_connect @redpanda_connect.processor def transform_message(msg: redpanda_connect.Message) -> redpanda_connect.Message: # Your transformation logic here msg.payload = msg.payload.upper() return msg if __name__ == "__main__": logging.basicConfig(level=logging.INFO) asyncio.run(redpanda_connect.processor_main(transform_message)) ``` ### [](#plugin-configuration-structure)Plugin configuration structure Each plugin requires a configuration file with the following structure: ```yaml name: summary: command: ["", ""] type: fields: [] # Configuration fields (if any) ``` ### [](#message-handling)Message handling The Python SDK provides access to message properties: - `msg.payload` - Message content - `msg.metadata` - Message metadata - Standard message transformation methods ## [](#next-steps)Next steps - Explore the [example plugins](https://github.com/redpanda-data/connect/tree/main/public/plugin/python/examples) in the Redpanda Connect repository - Join the [Redpanda Community Slack](https://redpanda.com/slack) to discuss plugin development - Review the [Redpanda Connect source code](https://github.com/redpanda-data/connect) for advanced use cases ## [](#related-topics)Related topics - [Components overview](https://docs.redpanda.com/connect/components/about/) - [Configuration overview](https://docs.redpanda.com/connect/configuration/about/) - [Getting started with Redpanda Connect](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) ## Suggested labs - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 412: Reference **URL**: https://docs.redpanda.com/connect/reference.md --- # Reference --- title: Reference latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: index page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/reference/pages/index.adoc description: The following topics provide reference material for Redpanda Connect. page-git-created-date: "2024-11-14" page-git-modified-date: "2024-11-14" --- - [Redpanda Connect Chart Specification](k-connect-helm-spec/) Find the default values and descriptions of settings in the Redpanda Connect Helm chart. --- # Page 413: Redpanda Connect Chart Specification **URL**: https://docs.redpanda.com/connect/reference/k-connect-helm-spec.md --- # Redpanda Connect Chart Specification --- title: Redpanda Connect Chart Specification latest-connect-version: 4.113.1 latest-operator-version: v26.2.4 latest-console-tag: v3.12.0 latest-redpanda-tag: v26.2.4 docname: k-connect-helm-spec page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: k-connect-helm-spec.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/reference/pages/k-connect-helm-spec.adoc description: Find the default values and descriptions of settings in the Redpanda Connect Helm chart. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-08-12" --- ![Version: 3.0.3](https://img.shields.io/badge/Version-3.0.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 4.42.0](https://img.shields.io/badge/AppVersion-4.42.0-informational?style=flat-square) Redpanda Connect is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads. This Helm chart deploys a Redpanda Connect instance in either streams mode or standalone. This page describes the contents of the chart’s [`values.yaml` file](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/values.yaml). Each of the settings is listed and described on this page, along with any default values. For instructions on how to install and use the chart, including how to override and customize the chart’s values, refer to the [deployment documentation](https://docs.redpanda.com/connect/get-started/quickstarts/helm-chart/). > 📝 **NOTE** > > To migrate to this chart from the [Benthos Helm chart](https://github.com/redpanda-data/redpanda-connect-helm-chart), see the [migration guide in this GitHub repository](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/MIGRATION_FROM_BENTHOS.md). * * * Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2) ## [](#source-code)Source Code - [https://github.com/redpanda-data/helm-charts](https://github.com/redpanda-data/helm-charts) ## [](#settings)Settings ### [](#affinity)[affinity](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=affinity) Affinity rules for scheduling Pods. **Default:** `{}` ### [](#args)[args](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=args) Override the default arguments for the entrypoint command. **Default:** `[]` ### [](#autoscaling)[autoscaling](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling) Configuration for Horizontal Pod Autoscaling for the Redpanda Connect Deployment. **Default:** {"enabled":false,"maxReplicas":12,"metrics":\[{"resource":{"name":"cpu","target":{"averageUtilization":80,"type":"Utilization"}},"type":"Resource"}\],"minReplicas":1} ### [](#autoscaling-enabled)[autoscaling.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.enabled) Enable Horizontal Pod Autoscaling. **Default:** `false` ### [](#autoscaling-maxreplicas)[autoscaling.maxReplicas](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.maxReplicas) Maximum number of replicas if autoscaling is enabled. **Default:** `12` ### [](#autoscaling-metrics)[autoscaling.metrics](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.metrics) List of metrics to use for scaling, such as CPU utilization. **Default:** \[{"resource":{"name":"cpu","target":{"averageUtilization":80,"type":"Utilization"}},"type":"Resource"}\] ### [](#autoscaling-minreplicas)[autoscaling.minReplicas](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.minReplicas) Minimum number of replicas if autoscaling is enabled. **Default:** `1` ### [](#command)[command](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=command) Override the default entrypoint command of the container. **Default:** `[]` ### [](#commonlabels)[commonLabels](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=commonLabels) Additional labels to apply to all resources created by this chart. **Default:** `{}` ### [](#config)[config](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=config) Configuration for `/redpanda-connect.yaml` file. **Default:** `{}` ### [](#deployment-annotations)[deployment.annotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.annotations) Set Annotations on the Redpanda Connect Deployment. **Default:** `{}` ### [](#deployment-livenessprobe)[deployment.livenessProbe](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe) Configuration for the liveness probe that checks if the container is running. **Default:** {"failureThreshold":3,"httpGet":{"path":"/ping","port":"http"},"periodSeconds":5,"successThreshold":1,"timeoutSeconds":2} ### [](#deployment-livenessprobe-failurethreshold)[deployment.livenessProbe.failureThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.failureThreshold) Number of consecutive failures before Pod is restarted. **Default:** `3` ### [](#deployment-livenessprobe-httpget)[deployment.livenessProbe.httpGet](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.httpGet) HTTP GET request configuration for the liveness probe. **Default:** {"path":"/ping","port":"http"} ### [](#deployment-livenessprobe-periodseconds)[deployment.livenessProbe.periodSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.periodSeconds) Frequency (in seconds) of liveness checks. **Default:** `5` ### [](#deployment-livenessprobe-successthreshold)[deployment.livenessProbe.successThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.successThreshold) Number of consecutive successes required to mark as healthy. **Default:** `1` ### [](#deployment-livenessprobe-timeoutseconds)[deployment.livenessProbe.timeoutSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.timeoutSeconds) Timeout (in seconds) for each liveness check. **Default:** `2` ### [](#deployment-podannotations)[deployment.podAnnotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.podAnnotations) Annotations to add to Redpanda Connect Pods. **Default:** `{}` ### [](#deployment-podlabels)[deployment.podLabels](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.podLabels) Labels to add to Redpanda Connect Pods. **Default:** `{}` ### [](#deployment-readinessprobe)[deployment.readinessProbe](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe) Configuration for the readiness probe that checks if the container is ready to accept traffic. **Default:** {"failureThreshold":1,"httpGet":{"path":"/ready","port":"http"},"periodSeconds":5,"successThreshold":1,"timeoutSeconds":2} ### [](#deployment-readinessprobe-failurethreshold)[deployment.readinessProbe.failureThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.failureThreshold) Number of consecutive failures before marking Pod as unhealthy. **Default:** `1` ### [](#deployment-readinessprobe-httpget)[deployment.readinessProbe.httpGet](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.httpGet) HTTP GET request configuration for the readiness probe. **Default:** {"path":"/ready","port":"http"} ### [](#deployment-readinessprobe-periodseconds)[deployment.readinessProbe.periodSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.periodSeconds) Frequency (in seconds) of readiness checks. **Default:** `5` ### [](#deployment-readinessprobe-successthreshold)[deployment.readinessProbe.successThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.successThreshold) Number of consecutive successes before marking Pod as healthy. **Default:** `1` ### [](#deployment-readinessprobe-timeoutseconds)[deployment.readinessProbe.timeoutSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.timeoutSeconds) Timeout (in seconds) for each readiness check. **Default:** `2` ### [](#deployment-replicacount)[deployment.replicaCount](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.replicaCount) Number of Replicas for the Redpanda Connect Deployment. **Default:** `1` ### [](#deployment-restartpolicy)[deployment.restartPolicy](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.restartPolicy) Restart policy for containers in the Pod. **Default:** `"Always"` ### [](#deployment-rolloutconfigmap)[deployment.rolloutConfigMap](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.rolloutConfigMap) Rollout the deployment on ConfigMap changes. **Default:** `true` ### [](#deployment-terminationgraceperiodseconds)[deployment.terminationGracePeriodSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.terminationGracePeriodSeconds) Time in seconds for Pod termination grace period. **Default:** `60` ### [](#env)[env](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=env) Environment variables to pass to the Redpanda Connect container. **Default:** `[]` ### [](#envfrom)[envFrom](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=envFrom) Names of Secrets or ConfigMaps that contain environment variables to pass to the Redpanda Connect container. **Default:** `[]` ### [](#extravolumemounts)[extraVolumeMounts](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=extraVolumeMounts) Additional volume mounts for the container. **Default:** `[]` ### [](#extravolumes)[extraVolumes](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=extraVolumes) Additional volumes to add to the Pod. **Default:** `[]` ### [](#fullnameoverride)[fullnameOverride](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=fullnameOverride) Override for the full name template. **Default:** `""` ### [](#http)[http](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http) Configuration for the HTTP server used to expose Redpanda Connect endpoints. **Default:** {"address":"0.0.0.0:4195","cors":{"enabled":false},"debug\_endpoints":false,"enabled":true,"root\_path":"/redpanda-connect"} ### [](#http-address)[http.address](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.address) Bind Address for the HTTP server. **Default:** `"0.0.0.0:4195"` ### [](#http-cors-enabled)[http.cors.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.cors.enabled) Enable CORS support for HTTP server. **Default:** `false` ### [](#http-debug_endpoints)[http.debug_endpoints](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.debug_endpoints) Enable debug endpoints for the HTTP server. **Default:** `false` ### [](#http-enabled)[http.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.enabled) Enable HTTP server. **Default:** `true` ### [](#http-root_path)[http.root_path](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.root_path) Root path for HTTP API endpoints. **Default:** `"/redpanda-connect"` ### [](#image)[image](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image) Configuration for the Redpanda Connect container image. **Default:** {"pullPolicy":"IfNotPresent","repository":"docker.redpanda.com/redpandadata/connect","tag":""} ### [](#image-pullpolicy)[image.pullPolicy](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image.pullPolicy) Image pull policy, such as `IfNotPresent` or `Always`. **Default:** `"IfNotPresent"` ### [](#image-repository)[image.repository](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image.repository) Docker repository for the Redpanda Connect image. **Default:** "docker.redpanda.com/redpandadata/connect" ### [](#image-tag)[image.tag](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image.tag) Override for the default image tag (uses chart `appVersion` if empty). **Default:** `""` ### [](#imagepullsecrets)[imagePullSecrets](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=imagePullSecrets) List of Secrets for private Docker registry authentication. **Default:** `[]` ### [](#ingress-annotations)[ingress.annotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.annotations) Annotations for the Ingress resource. **Default:** `{}` ### [](#ingress-classname)[ingress.className](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.className) Ingress Class name for the Ingress resource. **Default:** `""` ### [](#ingress-enabled)[ingress.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.enabled) Enable Ingress resource creation. **Default:** `false` ### [](#ingress-hosts)[ingress.hosts](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.hosts) List of host definitions for the Ingress resource. **Default:** `[]` ### [](#ingress-tls)[ingress.tls](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.tls) TLS configuration for Ingress resource. **Default:** `[]` ### [](#initcontainers)[initContainers](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=initContainers) Additional init containers to add to the Deployment. **Default:** `[]` ### [](#nameoverride)[nameOverride](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=nameOverride) **Default:** `""` ### [](#nodeselector)[nodeSelector](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=nodeSelector) Node selector for scheduling Pods. **Default:** `{}` ### [](#poddisruptionbudget)[podDisruptionBudget](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=podDisruptionBudget) Configuration for the PodDisruptionBudget, which controls voluntary evictions of Redpanda Connect Pods. **Default:** `{"enabled":false}` ### [](#poddisruptionbudget-enabled)[podDisruptionBudget.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=podDisruptionBudget.enabled) Enable a PodDisruptionBudget for the Deployment. **Default:** `false` ### [](#podsecuritycontext)[podSecurityContext](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=podSecurityContext) Security context to be applied at the Pod level. **Default:** `{}` ### [](#resources)[resources](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=resources) Resource limits and requests for the container. **Default:** `{}` ### [](#securitycontext)[securityContext](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=securityContext) Security context to be applied to containers in the Pod. **Default:** `{}` ### [](#service)[service](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service) Configuration for the Kubernetes Service that exposes Redpanda Connect. **Default:** {"extraPorts":null,"name":"http","port":80,"protocol":"TCP","targetPort":"http","type":"ClusterIP"} ### [](#service-extraports)[service.extraPorts](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.extraPorts) Additional ports to expose through the Service. **Default:** `nil` ### [](#service-name)[service.name](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.name) Name of the Kubernetes Service. **Default:** `"http"` ### [](#service-port)[service.port](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.port) Service port for accessing the application. **Default:** `80` ### [](#service-protocol)[service.protocol](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.protocol) Protocol to use for the Service, for example TCP or UDP. **Default:** `"TCP"` ### [](#service-targetport)[service.targetPort](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.targetPort) Target port on the container. **Default:** `"http"` ### [](#service-type)[service.type](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.type) Type of Kubernetes Service, for example ClusterIP, NodePort, or LoadBalancer. **Default:** `"ClusterIP"` ### [](#serviceaccount)[serviceAccount](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount) Configuration for the Kubernetes ServiceAccount associated with the Redpanda Connect Pods. **Default:** {"annotations":{},"create":true,"name":""} ### [](#serviceaccount-annotations)[serviceAccount.annotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount.annotations) Annotations to add to the ServiceAccount. **Default:** `{}` ### [](#serviceaccount-create)[serviceAccount.create](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount.create) Specify whether a ServiceAccount should be created. **Default:** `true` ### [](#serviceaccount-name)[serviceAccount.name](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount.name) Name of the ServiceAccount to use. If not set and `create` is `true`, a name is autogenerated. **Default:** `""` ### [](#servicemonitor)[serviceMonitor](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor) Configuration for the ServiceMonitor used by Prometheus to scrape Redpanda Connect metrics. **Default:** {"enabled":false,"interval":"10s","scheme":"http"} ### [](#servicemonitor-enabled)[serviceMonitor.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor.enabled) Enable the creation of a ServiceMonitor for Prometheus monitoring. **Default:** `false` ### [](#servicemonitor-interval)[serviceMonitor.interval](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor.interval) Scrape interval for Prometheus monitoring. **Default:** `"10s"` ### [](#servicemonitor-scheme)[serviceMonitor.scheme](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor.scheme) HTTP Scheme to use for scraping. **Default:** `"http"` ### [](#streams)[streams](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams) Configuration settings for the streams mode feature in Redpanda Connect. **Default:** {"api":{"enable":true},"enabled":false,"streamsConfigMap":""} ### [](#streams-api-enable)[streams.api.enable](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams.api.enable) Enable the streams API. **Default:** `true` ### [](#streams-enabled)[streams.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams.enabled) Enable streams mode. **Default:** `false` ### [](#streams-streamsconfigmap)[streams.streamsConfigMap](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams.streamsConfigMap) Name of the ConfigMap that contains stream configuration files. **Default:** `""` ### [](#telemetry)[telemetry](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=telemetry) **Default:** `true` ### [](#tolerations)[tolerations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=tolerations) Tolerations for scheduling Pods onto nodes with matching taints. **Default:** `[]` ### [](#topologyspreadconstraints)[topologySpreadConstraints](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=topologySpreadConstraints) **Default:** `[]` ### [](#updatestrategy)[updateStrategy](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=updateStrategy) Update strategy for the Deployment, for example `RollingUpdate`. **Default:** `{}` ### [](#watch)[watch](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=watch) Experimental feature: Watch config files for changes and automatically apply them without needing to rollout the Deployment. Make sure that `deployment.rolloutConfigMap` is `false`. **Default:** `false` ---