What’s New in Redpanda Connect
This topic includes new content added from version 4.29.0 onwards.
For a full list of product updates, and details of updates prior to version 4.29.0, see the Redpanda Connect changelog. For new Cloud-specific features, see What’s New in Redpanda Cloud.
Version 4.60.0
Component updates
The 4.60.0 release adds support for the following fields:
-
NATS components
-
tls_handshake_first
- Enhanced TLS configuration option for NATS connections that require the initial TLS handshake to be performed before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation.
-
This field is available in the following NATS components:
Version 4.59.0
Redpanda Connect in Redpanda Cloud: GA
Redpanda Connect is now generally available (GA) in all Redpanda Cloud clusters: BYOC (including BYOVPC/BYOVNet), Dedicated, and Serverless.
Redpanda Connect has been GA in Redpanda Self-Managed clusters since version 4.29.0.
Version 4.58.2
This release introduces an experimental Slack output and expands support for automatic topic creation across Redpanda-related components.
New components
-
slack_reaction
(experimental): Adds or removes emoji reactions from Slack messages by channel ID and timestamp.
Component updates
This release adds support for the allow_auto_topic_creation
field in the following components:
-
redpanda
configuration block. -
kafka_franz
output. -
redpanda
output. -
redpanda_migrator
output.
This field allows components to create topics automatically when publishing data, if the underlying Kafka/Redpanda broker is configured to allow it.
Version 4.57.0
This release adds support for the following fields:
-
protobuf.use_proto_names
,protobuf.use_enum_numbers
,protobuf.emit_unpopulated
, andprotobuf.emit_default_values
in theschema_registry_decode
processor.
Version 4.56.0
This release introduces a dynamic plugins framework. It enables you to create and load custom plugins at runtime using any programming language that supports gRPC, with initial SDKs for Go and Python.
New components
-
gcp_spanner_cdc
(beta): Streams change data from a Google Cloud Spanner instance.
Component updates
This release adds support for the following fields:
-
scope
in thecouchbase
cache. -
poll_interval
andmax_yield_batch_bytes
in theredpanda_migrator_offsets
input. -
consumer_group_offsets_poll_interval
in theredpanda_migrator_bundle
input. -
input_bundle_label
in theredpanda_migrator_bundle
output. -
object_canned_acl
in theaws_s3
output. -
history
,max_tool_calls
, andtools
in thegcp_vertex_ai_chat
processor.
Version 4.55.1
This release adds support for the following fields:
-
is_serverless
in theredpanda_migrator
output.
Version 4.55.0
This release introduces a new beta Redpanda cache component and new fields across several inputs and outputs.
New components
-
redpanda
(beta): A new cache that stores key/value pairs in a compacted Redpanda topic.
Component updates
This release adds support for the following fields:
-
steal_grace_period
in theaws_kinesis
input. -
max_yield_batch_bytes
in:-
redpanda_migrator
input -
redpanda_migrator
output
-
extras
in thesentry_capture
processor.
Removed components
The deprecated elasticsearch
output has been removed. Use the elasticsearch_v8
output instead.
New Bloblang features
-
bitwise_and
,bitwise_or
, andbitwise_xor
methods added for performing bitwise operations on integers.
Version 4.54.0 to 4.54.1
These releases introduce the qdrant
processor, which allows you to query items within a Qdrant collection and filter the returned results.
Component updates
These releases add:
-
The
topic_lag_refresh_period
field to thekafka_franz
andockam_kafka
inputs. -
The
cache_duration
field to theschema_registry_decode
processor. -
The
client_auth
field to thesocket_server
input.
You can also now use Bloblang to generate a name-based UUID (version 5) for a given string.
Version 4.53.0
This release introduces three new Google Drive processors:
-
google_drive_search
processor: Searches Google Drive for files that match a specified query and emits the results as a batch of messages. -
google_drive_download
processor: Downloads files from Google Drive that contain matching file IDs. -
google_drive_list_labels
processor: Lists labels for files on a Google Drive.
The release also adds the cohere_rerank
processor, which sends document strings to the Cohere API, and returns them ranked by their relevance to a specified query.
Component updates
This release adds:
-
The
transaction_isolation_level
field to the following inputs:kafka_franz
,ockam_kafka
,redpanda
,redpanda_common
,redpanda_migrator
. -
The
request_timeout_overhead
andconn_idle_timeout
fields to the following components:kafka_franz
input,redpanda
input,redpanda_migrator
input,kafka_franz
output,redpanda
output,redpanda_migrator
output,redpanda_migrator_offsets
output,redpanda
configuration service. -
The
start_offset
field to the following components:kafka_franz
input,ockam_kafka
input,redpanda
input,redpanda_common
input,redpanda_migrator
input. -
The
topic_prefix
field to theredpanda_migrator
output, and theoffset_topic_prefix
field to theredpanda_migrator_offsets
output, to more easily identify migrated topics. -
The
use_enum_numbers
field to theprotobuf
processor. -
The
dimensions
field to thecohere_embeddings
processor. -
The
region
,endpoint
, andcredentials
fields to thedynamodb
section of theaws_kinesis
input. -
A
tools
section to thecohere_chat
processor to enhance response generation with agentic-like behavior.
Deprecations
This release deprecates the start_from_oldest
field from the following components: kafka_franz
input, ockam_kafka
input, redpanda
input, redpanda_common
input, redpanda_migrator
input. Now, you must use the start_offset
field instead, which defaults to earliest
. Redpanda recommends upgrading to benefit from this update.
Version 4.52.0
This release introduces three new Slack components:
-
slack_users
input: Reads the full profiles of users in a Slack organization. -
slack_thread
processor: Reads all messages in a Slack thread. -
slack_post
output: Posts messages to Slack channel.
Version 4.51.0
Highlights of this update include the introduction of three new components that you could use as part of an AI toolchain:
You can also now install a FIPS-compliant version of Redpanda Connect (for RHEL/Debian Ubuntu).
Component updates
This release adds:
-
The
private_key
field to thesftp
input and output -
The
history
field to theollama_chat
andopenai_chat_completion
processors, which allows you to include historical messages in a chat request. -
The
handle_logical_types
fields to theparquet_decode
processor. -
The
aggregate
operation to themongodb
processor, which provides support for aggregation pipelines. -
The
headers
option to thetype
field in theamqp_0_9
output.
Version 4.50.0
This release introduces agentic behavior to the openai_chat_completion
processor. You can now specify external tools to enrich the responses of OpenAI large language models.
Component updates
This release:
-
Adds a set of
tools
fields to theopenai_chat_completion
processor. -
Extends support for FLOAT values in the
snowflake_streaming
output to include special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity).
Redpanda Connect also now supports the unicode_segments
Bloblang method, which splits a string into segments using Unicode text segmentation rules.
Version 4.49.1
This release introduces further updates to the postgres_cdc
input and snowflake_streaming
output.
Component updates
This release adds:
-
Two new metrics to the
snowflake_streaming
output:snowflake_register_latency_ns
andsnowflake_commit_latency_ns
. -
An
ignore_nulls
option for schema evolution in thesnowflake_streaming
output. -
The
translate_ids
field to theschema_registry
output. -
The
translate_schema_ids
field to theredpanda_migrator_bundle
output.
Also, the translate_schema_ids
field in the redpanda_migrator
output now defaults to false
.
Deprecations
This release deprecates the snapshot_memory_factor
field from the postgres_cdc
input. Now, you must set a batch size, which defaults to 1000
messages. Redpanda recommends upgrading to benefit from this update.
Version 4.48.1
This release introduces a change data capture (CDC) input for MongoDB on Cloud and Self-Managed Redpanda Connect, along with improvements to the postgres_cdc
input and snowflake_streaming
output.
Component updates
This release adds:
-
The
heartbeat_interval
field to thepostgres_cdc
input, which helps manage the size of the Write-Ahead Log when you subscribe to data changes from tables with low activity. -
The
commit_timeout
field to thesnowflake_streaming
output. -
The
url
field to thesnowflake_streaming
output, which allows you to override the default connection URL. -
The
is_high_watermark
field to theredpanda_migrator_offsets
output. -
The
kafka_is_high_watermark
metadata field to theredpanda_migrator_offsets
input.
Other changes:
-
You can now load an Enterprise Edition license using the
REDPANDA_LICENSE
environment variable. -
The
postgres_cdc
input no longer adds the prefix_rs
to the replication slots it creates. -
The
clickhouse
driver is now supported in Redpanda Cloud for all components with the prefixsql_
. -
The
redpanda migrator
output now attempts to transfer access control lists for topics that already exist on the destination broker.
Version 4.47.0 to 4.47.1
This release introduces:
-
The
elasticsearch_v8
output: A certified, up-to-date alternative to the existingelasticsearch
output. -
The
crash
processor: A tool for detecting unhandled pipeline errors during development and testing.
Component updates
This release adds:
-
The
arguments
field to theamqp_0_9
input and output. -
The
mapping
field (for Avro messages) to theschema_registry_decode
processor. -
The
disable_http2
field to thehttp_client
input, output, andhttp
processor. -
The
retry_on_conflict
field to theelasticsearch
output.
Redpanda Connect also now supports the uuid_v7
Bloblang function, which is useful for scenarios when the exact timing of events is important, such as during data migration or replication.
Version 4.46.0
This release introduces a change data capture (CDC) input for MySQL, and enhancements to the Bloblang playground, making it easier to test and debug your pipeline configurations. The enhancements include:
-
A Run button in Bloblang code snippets, which opens each snippet in the playground with a pre-filled input and mapping ready for testing.
-
A Get help from AI button, which appears in the playground when an error occurs.
Component updates
This release adds:
-
The
instance_id
field to thekafka
,kafka_franz
,ockam_kafka
,redpanda
,redpanda_common
, andredpanda_migrator
inputs. -
The
rebalance_timeout
,session_timeout
, andheartbeat_interval
fields to thekafka_franz
,ockam_kafka
,redpanda
,redpanda_common
, andredpanda_migrator
inputs. -
The
preserve_logical_types
andraw_unions
fields (for Avro messages), and aschema_id
metadata field to theschema_registry_decode
processor. -
The
unchanged_toast_value
field to thepostgres_cdc
input. -
A
processors
field to theschema_evolution
object in thesnowflake_streaming
output. -
New default values to the
max_message_bytes
andbroker_write_max_bytes
fields in thekafka_franz
,ockam_kafka
,redpanda
,redpanda_migrator
, andredpanda_migrator_offsets
outputs, using IEC instead of SI units.
Other changes:
-
The
postgres_cdc
input no longer supports PostgreSQL 10 or 11.
Version 4.45.0
Highlights of this update include the beta release of two components for Self-Managed and Cloud Redpanda Connect, and a new cookbook that explains how to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. You can also now share links to code snippets in the Bloblang playground using the Share link button.
Components for Cloud and Self-Managed Redpanda Connect (beta)
-
ollama_moderation
processor (Self-Managed, Cloud) -
redpanda_migrator_offsets
input (Self-Managed, Cloud)
Component updates
This release adds:
-
The
max_outstanding_messages
andmessage_timeout
fields to theaws_sqs
input. -
The
content_type
field to theamqp_1
output. -
The
offset_token
field to thesnowflake_streaming
output. -
The
save_prompt_metadata
,max_tool_calls
fields andtools
object to theollama_chat
processor. -
The
offset_topic
,offset_group
,offset_partition
,offset_commit_timestamp
andoffset_metadata
fields to theredpanda_migrator_offsets
output. -
The
topic_lag_refresh_period
field to theredpanda
andredpanda_common
inputs. -
The
redpanda_lag
metric to theredpanda
andredpanda_common
inputs. -
The
fetch_max_wait
field to thekafka_franz
,ockam_kafka
,redpanda
,redpanda_common
, andredpanda_migrator
inputs. -
The
kafka_lag
metadata field to theredpanda
andredpanda_common
inputs. -
The
avro_schema
andavro_schema_fingerprint
metadata fields to theavro
scanner. -
The
label
field to template test definitions. -
The
bloblang
scalar type to template fields.
Redpanda Connect now supports:
-
Interpolation functions for channel and table names in the
snowflake_streaming
output. -
Calls to external tools when using the
ollama_chat
processor. -
The execution of multiple database queries as transactions when using the
sql_raw
input, output and processor. -
Using the
label
metadata field within a template mapping to retrieve the template componentlabel
set in the corresponding configuration. -
The following Bloblang functions for error handling:
error_source_name
,error_source_label
, anderror_source_path
.
Version 4.44.0
This release extends the functionality of the openai_chat_completion
processor to include the following fields: max_tokens
, temperature
, user
, top_p
, frequency_penalty
, presence_penalty
, seed
, and stop
.
Version 4.43.1
This release introduces support for trial licenses for Self-Managed Redpanda Connect.
Version 4.43.0
This release renames the pg_stream
input to postgres_cdc
.
The postgres_cdc
input also no longer emits mode
metadata, but sets the operation
metadata field to read
when a snapshot of a database is processed.
Version 4.42.0
This release introduces a Bloblang playground where you can try out your Bloblang syntax and get immediate feedback. It also updates Redpanda Connect licensing checks for Redpanda Connect. Starting from version 4.42.0, you must add your Enterprise Edition license to Redpanda Connect before using any Enterprise connectors.
Component updates
Redpanda Connect now supports:
-
A Google Cloud
spanner
driver for all SQL plugins. -
The following complex data types for the
pg_stream
input: JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]. -
Parquet files for the
bigquery
output.
This release also adds an exists
operator to the cache
processor.
Version 4.41.0
This release adds the max_records_per_request
field to the aws_sqs
output.
Packaging update
Starting from version 4.41.0, the -cgo
suffixed Docker image is no longer available due to low demand and the unacceptable cadence with which the image base (Debian) receives security updates.
To create your own CGO builds, run the following command:
CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect.
Version 4.40.0
This update includes a beta release of:
-
The
pg_stream
input, which captures data changes made to a PostgreSQL database in real time and streams them to Redpanda Connect. -
The
benchmark
processor, which provides message throughput statistics for a data pipeline.
It also introduces a new Bloblang method for timestamps.
Component updates
This release adds:
-
The
metadata_max_age
field to theredpanda_migrator_offsets
output. -
The
timestamp_ms
field to the following outputs:ockam_kafka
,kafka
,kafka_franz
,redpanda
,redpanda_common
,redpanda_migrator
. -
The
kafka_timestamp_ms
metadata field, which uses a Unix timestamp with millisecond precision, to the following inputs:kafka
,kafka_franz
,redpanda
,redpanda_common
,redpanda_migrator
. -
The
translate_schema_ids
andschema_registry_output_resource
to theredpanda_migrator
output. -
The
backfill_dependencies
andinput_resource
fields to theschema_registry
output. -
Options to optimize the build of the data output from the
snowflake_streaming
output.
Other changes:
-
The following components now use the Franz Kafka Schema Registry client library:
schema_registry
input and output, and theschema_registry_encode
andschema_registry_decode
processors. -
The
snowflake_streaming
output can now automatically create tables in Snowflake, when theschema_evolution.enabled
field is set totrue
.
Deprecations
Deprecated field: kafka_timestamp
metadata field
Affected components:
Replaced by: kafka_timestamp_ms
metadata field
Deprecated field: timestamp
Affected components:
Replaced by: timestamp_ms
field
Action required: Modify all systems that process these fields to ensure compatibility with the new field names.
Version 4.39.0
Highlights of this update include the beta release of three new components for Self-Managed and Cloud, including a new snowflake_streaming
output. This release also includes additions to the redpanda
family of components and introduces new features for Self-Managed.
Additional redpanda
components (beta)
-
redpanda
(input and output): Consume message data from or send message data to Kafka brokers. -
redpanda_common
(input and output): Consume message data from or send message data to a Redpanda Kafka broker, using credentials from a commonredpanda
configuration block (Self-Managed only).
Features in Self-Managed
You can now:
-
Create an allow list or deny list for a Redpanda Connect instance
-
Use the
rpk connect
CLI flag--secrets
to look up secrets on a remote system -
Use the
rpk connect
CLI flag--disable-telemetry
to block the telemetry service
Component updates
This release adds:
-
The
follow_redirects
field to thehttp
processor. -
The following fields to the
redpanda
configuration service:-
metadata_max_age
-
partitioner
-
idempotent_write
-
compression
-
broker_write_max_bytes
-
-
The
metadata_max_age
field to thekafka_franz
output. -
The following fields to the
kafka_franz
andockam_kafka
inputs:-
fetch_max_bytes
-
fetch_min_bytes
-
fetch_max_partition_bytes
-
-
The
url
field of theaws_sqs
output now supports interpolation functions.
Version 4.38.0
This update includes the beta release of two new outputs for Cloud and Self-Managed, and the introduction of a telemetry service, which collects and sends anonymized usage statistics to Redpanda.
New tooling
A new Helm chart for deploying Redpanda Connect in Kubernetes is now available. This chart simplifies the deployment and configuration of Redpanda Connect, allowing you to manage and scale Redpanda Connect instances in Kubernetes environments.
To get started with the Helm chart, see Get Started with the Redpanda Connect Helm Chart.
Or, see the release notes for the chart on GitHub.
New outputs (beta)
-
azure_data_lake_gen2
(certified) -
timeplus
(community)
Component updates
This release adds:
-
The
checksum_algorithm
to theaws_s3
output. -
The
private_key
field to thesnowflake_put
output. -
The
nkey
field to the following components:
Other changes:
-
The
collection
field of themongodb
output now supports interpolation functions. -
All components that used the deprecated
count
function, with a default value for thepath
field, now use thecounter
function.If you have multiple components that execute a mapping using the deprecated count
function with the original default value, and these counters need to cascade, there may be a change in behavior. To avoid this, explicitly define a value for thepath
field.
Version 4.37.0
Redpanda Connect has new bundled Redpanda Migrator components, more AI processors for Self-Managed and Cloud platforms, and three new outputs.
Redpanda Migrator components (beta)
You can now move workloads from any Apache Kafka system to Redpanda using the redpanda_migrator_bundle
input and output.
For full details, see the Redpanda Migrator Cookbook for Self-Managed and Cloud.
Component updates
This release adds:
-
The
metadata_max_age
field tokafka_franz
andredpanda_migrator
inputs. -
The
fetch_in_order
field to theschema_registry
input. -
The
encrypted_fields
fields to theockam_kafka
input and output.
Version 4.36.0
You can now use the replication_factor_override
and replication_factor
fields of the redpanda_migrator
input and output to set a replication factor for migrated topics.
Versions 4.35.1 to 4.35.3
Redpanda Connect is now fully integrated with the Redpanda Cloud platform. This release adds Azure and GCP components to the growing list of available components. For a full list of components supported in Redpanda Cloud, see the Connector Catalog.
Version 4.35.0
Redpanda Connect now includes beta versions of Redpanda Migrator components:
For full details, see the Redpanda Migrator Cookbook for Self-Managed and Cloud.
Component updates
This release adds:
-
OAuth, basic authentication, and JSON Web Token fields to the
schema_registry
input and output. -
The
job_project
field to thegcp_bigquery
output. -
The
broker_write_max_bytes
field to thekafka_franz
andockam_kafka
outputs.
Version 4.34.0
Redpanda Connect now includes beta versions of two new AI processors for Cloud and Self-Managed platforms:
Version 4.33.0
Redpanda Connect now includes beta versions of four new components.
For Self-Managed only
You can also use the new Bloblang vector
method to create vector embeddings for SQL databases.
Component updates
This release adds:
-
The
credentials_json
field to all GCP components. -
The
content_md5
field to theaws_s3
output. -
The
send_ack
field to thenats
input.
Version 4.32.1
Version 4.31.0
Redpanda Connect now includes beta versions of two new components:
-
pinecone
output - Redpanda Connect’s first AI component
Component updates
This release adds:
-
Custom TLS fields for the
splunk
input andsplunk_hec
output. -
The
timestamp
field tokafka
andkafka_franz
outputs. -
The
status_topic
field to theredpanda
component.
Version 4.30.0 to 4.30.1
Redpanda Connect now includes a beta version of the splunk
input.
Version 4.29.0
Redpanda Connect now separates packages into an enterprise bundle and a free bundle, which contains community and selected certified connectors. To learn more about licensing and support levels, see Connector Support Levels.
Component updates
This release adds:
-
The
auth.oauth2.scope
field to thepulsar
input and output. -
The
subscription_initial_position
field to thepulsar
input.