What’s New in Redpanda Connect

NATS components

nats input

nats_jetstream input

nats_kv input

nats_stream input

nats output

nats_jetstream output

nats_kv output

nats_stream output

nats_kv processor

nats_request_reply processor

nats_kv cache

SFTP input components

SFTP output components

GCP Pub/Sub output components

Redpanda cache components

slack_reaction (experimental): Adds or removes emoji reactions from Slack messages by channel ID and timestamp.

redpanda configuration block.

kafka_franz output.

redpanda output.

redpanda_migrator output.

gcp_spanner_cdc (beta): Streams change data from a Google Cloud Spanner instance.

scope in the couchbase cache.

poll_interval and max_yield_batch_bytes in the redpanda_migrator_offsets input.

consumer_group_offsets_poll_interval in the redpanda_migrator_bundle input.

input_bundle_label in the redpanda_migrator_bundle output.

object_canned_acl in the aws_s3 output.

history, max_tool_calls, and tools in the gcp_vertex_ai_chat processor.

redpanda (beta): A new cache that stores key/value pairs in a compacted Redpanda topic.

steal_grace_period in the aws_kinesis input.

max_yield_batch_bytes in:

extras in the sentry_capture processor.

bitwise_and, bitwise_or, and bitwise_xor methods added for performing bitwise operations on integers.

The topic_lag_refresh_period field to the kafka_franz and ockam_kafka inputs.

The cache_duration field to the schema_registry_decode processor.

The client_auth field to the socket_server input.

The transaction_isolation_level field to the following inputs: kafka_franz, ockam_kafka, redpanda, redpanda_common, redpanda_migrator.

The request_timeout_overhead and conn_idle_timeout fields to the following components: kafka_franz input, redpanda input, redpanda_migrator input, kafka_franz output, redpanda output, redpanda_migrator output, redpanda_migrator_offsets output, redpanda configuration service.

The start_offset field to the following components: kafka_franz input, ockam_kafka input, redpanda input, redpanda_common input, redpanda_migrator input.

The topic_prefix field to the redpanda_migrator output, and the offset_topic_prefix field to the redpanda_migrator_offsets output, to more easily identify migrated topics.

The use_enum_numbers field to the protobuf processor.

The dimensions field to the cohere_embeddings processor.

The region, endpoint, and credentials fields to the dynamodb section of the aws_kinesis input.

A tools section to the cohere_chat processor to enhance response generation with agentic-like behavior.

The private_key field to the sftp input and output

The history field to the ollama_chat and openai_chat_completion processors, which allows you to include historical messages in a chat request.

The handle_logical_types fields to the parquet_decode processor.

The aggregate operation to the mongodb processor, which provides support for aggregation pipelines.

The headers option to the type field in the amqp_0_9 output.

Adds a set of tools fields to the openai_chat_completion processor.

Extends support for FLOAT values in the snowflake_streaming output to include special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity).

Two new metrics to the snowflake_streaming output: snowflake_register_latency_ns and snowflake_commit_latency_ns.

An ignore_nulls option for schema evolution in the snowflake_streaming output.

The translate_ids field to the schema_registry output.

The translate_schema_ids field to the redpanda_migrator_bundle output.

The heartbeat_interval field to the postgres_cdc input, which helps manage the size of the Write-Ahead Log when you subscribe to data changes from tables with low activity.

The commit_timeout field to the snowflake_streaming output.

The url field to the snowflake_streaming output, which allows you to override the default connection URL.

The is_high_watermark field to the redpanda_migrator_offsets output.

The kafka_is_high_watermark metadata field to the redpanda_migrator_offsets input.

You can now load an Enterprise Edition license using the REDPANDA_LICENSE environment variable.

The postgres_cdc input no longer adds the prefix _rs to the replication slots it creates.

The clickhouse driver is now supported in Redpanda Cloud for all components with the prefix sql_.

The redpanda migrator output now attempts to transfer access control lists for topics that already exist on the destination broker.

The arguments field to the amqp_0_9 input and output.

The mapping field (for Avro messages) to the schema_registry_decode processor.

The disable_http2 field to the http_client input, output, and http processor.

The retry_on_conflict field to the elasticsearch output.

mysql_cdc input

The instance_id field to the kafka, kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs.

The rebalance_timeout, session_timeout, and heartbeat_interval fields to the kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs.

The preserve_logical_types and raw_unions fields (for Avro messages), and a schema_id metadata field to the schema_registry_decode processor.

The unchanged_toast_value field to the postgres_cdc input.

A processors field to the schema_evolution object in the snowflake_streaming output.

New default values to the max_message_bytes and broker_write_max_bytes fields in the kafka_franz, ockam_kafka, redpanda, redpanda_migrator, and redpanda_migrator_offsets outputs, using IEC instead of SI units.

The postgres_cdc input no longer supports PostgreSQL 10 or 11.

ollama_moderation processor (Self-Managed, Cloud)

redpanda_migrator_offsets input (Self-Managed, Cloud)

The max_outstanding_messages and message_timeout fields to the aws_sqs input.

The content_type field to the amqp_1 output.

The offset_token field to the snowflake_streaming output.

The save_prompt_metadata, max_tool_calls fields and tools object to the ollama_chat processor.

The offset_topic, offset_group, offset_partition, offset_commit_timestamp and offset_metadata fields to the redpanda_migrator_offsets output.

The topic_lag_refresh_period field to the redpanda and redpanda_common inputs.

The redpanda_lag metric to the redpanda and redpanda_common inputs.

The fetch_max_wait field to the kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs.

The kafka_lag metadata field to the redpanda and redpanda_common inputs.

The avro_schema and avro_schema_fingerprint metadata fields to the avro scanner.

The label field to template test definitions.

The bloblang scalar type to template fields.

Interpolation functions for channel and table names in the snowflake_streaming output.

Calls to external tools when using the ollama_chat processor.

The execution of multiple database queries as transactions when using the sql_raw input, output and processor.

Using the label metadata field within a template mapping to retrieve the template component label set in the corresponding configuration.

The following Bloblang functions for error handling: error_source_name, error_source_label, and error_source_path.

A Google Cloud spanner driver for all SQL plugins.

The following complex data types for the pg_stream input: JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[].

Parquet files for the bigquery output.

pg_stream input

benchmark processor

The metadata_max_age field to the redpanda_migrator_offsets output.

The timestamp_ms field to the following outputs: ockam_kafka, kafka, kafka_franz, redpanda, redpanda_common, redpanda_migrator.

The kafka_timestamp_ms metadata field, which uses a Unix timestamp with millisecond precision, to the following inputs: kafka, kafka_franz, redpanda, redpanda_common, redpanda_migrator.

The translate_schema_ids and schema_registry_output_resource to the redpanda_migrator output.

The backfill_dependencies and input_resource fields to the schema_registry output.

Options to optimize the build of the data output from the snowflake_streaming output.

The following components now use the Franz Kafka Schema Registry client library: schema_registry input and output, and the schema_registry_encode and schema_registry_decode processors.

The snowflake_streaming output can now automatically create tables in Snowflake, when the schema_evolution.enabled field is set to true.

snowflake_streaming output

timeplus input

spicedb_watch input

redpanda (input and output): Consume message data from or send message data to Kafka brokers.

redpanda_common (input and output): Consume message data from or send message data to a Redpanda Kafka broker, using credentials from a common redpanda configuration block (Self-Managed only).

Create an allow list or deny list for a Redpanda Connect instance

Use the rpk connect CLI flag --secrets to look up secrets on a remote system

Use the rpk connect CLI flag --disable-telemetry to block the telemetry service

The follow_redirects field to the http processor.

The following fields to the redpanda configuration service:

The metadata_max_age field to the kafka_franz output.

The following fields to the kafka_franz and ockam_kafka inputs:

The url field of the aws_sqs output now supports interpolation functions.

azure_data_lake_gen2 (certified)

timeplus (community)

The checksum_algorithm to the aws_s3 output.

The private_key field to the snowflake_put output.

The nkey field to the following components:

The collection field of the mongodb output now supports interpolation functions.

All components that used the deprecated count function, with a default value for the path field, now use the counter function.

aws_bedrock_embeddings

cohere_chat

cohere_embeddings

gcp_vertex_ai_embeddings

couchbase (community)

cypher (community)

questdb (certified)

The metadata_max_age field to kafka_franz and redpanda_migrator inputs.

The fetch_in_order field to the schema_registry input.

The encrypted_fields fields to the ockam_kafka input and output.

OAuth, basic authentication, and JSON Web Token fields to the schema_registry input and output.

The job_project field to the gcp_bigquery output.

The broker_write_max_bytes field to the kafka_franz and ockam_kafka outputs.

schema_registry input and output

qdrant output

ockam_kafka input and output

The credentials_json field to all GCP components.

The content_md5 field to the aws_s3 output.

The send_ack field to the nats input.

openai_chat_completion

openai_embeddings

openai_image_generation

openai_speech

openai_transcription

openai_translation

ollama_chat

ollama_embeddings

Custom TLS fields for the splunk input and splunk_hec output.

The timestamp field to kafka and kafka_franz outputs.

The status_topic field to the redpanda component.

The auth.oauth2.scope field to the pulsar input and output.

The subscription_initial_position field to the pulsar input.