Connect

What’s New in Redpanda Connect

This topic includes new content added from version 4.29.0 onwards.

For a full list of product updates, and details of updates prior to version 4.29.0, see the Redpanda Connect changelog. For new Cloud-specific features, see What’s New in Redpanda Cloud.

Version 4.90.3

See the full release notes.

Salesforce connector refactoring

The salesforce processor (introduced in version 4.85.0) has been removed and replaced with dedicated input components that provide the same functionality with better separation of concerns. The salesforce_sink output remains available.

Component Type Status Description

salesforce

Inputs

stable

Executes SOQL queries against Salesforce and returns the results as messages.

salesforce_cdc

Inputs

stable

Captures change data from Salesforce objects using the Pub/Sub API.

salesforce_graphql

Inputs

stable

Executes GraphQL queries against Salesforce and returns the results as messages.

Migration guide

To migrate from the removed salesforce processor, use the appropriate input component based on your use case:

Previous processor feature Replacement

SOQL queries

salesforce input

Change Data Capture (CDC)

salesforce_cdc input

GraphQL queries

salesforce_graphql input

Version 4.90.1

See the full release notes.

Version 4.90.0

See the full release notes.

Component updates

This release adds the following new component:

Component Type Status Description

gcp_bigquery_write_api

Outputs

stable

Writes messages to a BigQuery table using the Storage Write API. This provides higher throughput and lower latency than the legacy streaming API or load jobs.

New field support

This release adds support for the following new fields:

Field Description Affected components

max_parallel_snapshot_tables

Specifies the number of tables that will be snapshotted in parallel.

Inputs:

Version 4.89.3

See the full release notes.

New field support

This release adds support for the following new fields:

Field Description Affected components

case_sensitive_columns

Controls how message field names are matched against table column names, and how column references in the partition spec are resolved.

Outputs:

Version 4.89.2

See the full release notes.

Version 4.89.1

See the full release notes.

Version 4.89.0

See the full release notes.

New field support

This release adds support for the following new fields:

Field Description Affected components

default_timestamp_unit

The precision used when encoding TIMESTAMP logical types.

Processors:

Version 4.88.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Status Description

arc

Outputs

stable

This output sends data to an Arc columnar analytical database using its high-performance MessagePack ingestion endpoint.

open_telemetry_collector

Metrics

stable

Exports Redpanda Connect metrics to one or more OpenTelemetry Collector endpoints over HTTP or gRPC for aggregation and onward export.

New field support

This release adds support for the following new fields:

Field Description Affected components

pdb_name

The name of the pluggable database (PDB) to monitor. When connecting to a CDB root, LogMiner output is scoped to this PDB via SRC_CON_NAME filtering and catalog queries use ALTER SESSION SET CONTAINER to switch context.

Inputs:

Deprecations and removals

The MCP (Model Context Protocol) server feature in Redpanda Connect has been deprecated and removed. This includes:

  • All MCP server documentation

  • The following CLI commands:

    • rpk connect mcp-server

    • rpk connect mcp-server init

    • rpk connect mcp-server lint

Version 4.87.0

See the full release notes.

New field support

This release adds support for the following new fields:

Field Description Affected components

checkpoint_cache_connection_string

An optional connection string for a remote Microsoft SQL Server to use for the checkpoint cache. When set, the checkpoint cache table is created on this remote server instead of the source database.

Inputs:

wallet_path

Path to the Oracle Wallet directory. When set, SSL is enabled automatically.

Inputs:

wallet_password

Password for the ewallet.p12 PKCS#12 wallet file. Only required when the wallet directory contains ewallet.p12 rather than cwallet.sso.

Inputs:

message_properties_message_id

Set the message-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string.

Outputs:

message_properties_correlation_id

Set the correlation-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string.

Outputs:

message_properties_subject

Set the subject property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

message_properties_reply_to

Set the reply-to property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

message_properties_group_id

Set the group-id property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

message_properties_group_sequence

Set the group-sequence property on outgoing AMQP messages. Must be a valid uint32 value.

Outputs:

message_properties_reply_to_group_id

Set the reply-to-group-id property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

message_properties_user_id

Set the user-id property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

message_properties_content_type

Set the content-type property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

message_properties_content_encoding

Set the content-encoding property on outgoing AMQP messages. This field supports Bloblang interpolation.

Outputs:

Version 4.86.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Status Description

string_split

Processors

stable

Splits a string by a delimiter into an array. Generally, using bloblang’s split method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic.

Version 4.85.0

See the full release notes.

Bloblang updates

This release adds the following new Bloblang capabilities:

  • Methods:

    • escape_url_path

      Escapes a string so it can be safely placed inside a URL path segment, replacing special characters with percent-encoded sequences.

    • parse_logfmt

      Parses a logfmt-encoded string into an object of key-value pairs.

    • unescape_url_path

      Unescapes a URL path segment, converting percent-encoded sequences back to their original characters.

Component updates

This release adds the following new components:

Component Type Description

oracledb_cdc

Inputs

Streams changes from an Oracle database for Change Data Capture (CDC).

salesforce_sink

Outputs

Consumes batches of messages and writes them to Salesforce.

salesforce

Processors

Fetches data from Salesforce based on input messages.

New field support

This release adds support for the following new fields:

Field Description Affected components

client_name

Set the client name for the Redis connection.

Caches:

Inputs:

Outputs:

Processors:

Rate-limits:

tables

List of table names to stream from. For single table mode, provide one table.

Inputs:

table_discovery_mode

Table discovery mode.

Inputs:

table_tag_filter

Multi-tag filter: 'key1:v1,v2;key2:v3,v4'. Matches tables with (key1=v1 OR key1=v2) AND (key2=v3 OR key2=v4).

Inputs:

table_discovery_interval

Interval for rescanning and discovering new tables when using tag or includelist mode. Set to 0 to disable periodic rescanning.

Inputs:

snapshot_mode

Snapshot behavior.

Inputs:

snapshot_segments

Number of parallel scan segments (1-10). Higher parallelism scans faster but consumes more RCUs.

Inputs:

snapshot_batch_size

Records per scan request during snapshot. Maximum 1000.

Inputs:

snapshot_throttle

Minimum time between scan requests per segment. Use this to limit RCU consumption during snapshot.

Inputs:

snapshot_deduplicate

Deduplicate records that appear in both snapshot and CDC stream. Requires buffering CDC events during snapshot.

Inputs:

snapshot_buffer_size

Maximum CDC events to buffer for deduplication (approximately 100 bytes per entry). If exceeded, deduplication is disabled and duplicates may be emitted.

Inputs:

create_stream

Whether to automatically create the stream if it doesn’t exist (requires the stream field to be set).

Inputs:

extract_tracing_map

A Bloblang mapping that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer.

Inputs:

id

The entry ID for the stream message. Allows function interpolations.

Outputs:

inject_tracing_map

A Bloblang mapping used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer.

Outputs:

sync_topic_interval

How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow.

Outputs:

commit_backoff

Control how frequently Snowflake is polled to check if data has been committed.

Outputs:

timeout

HTTP request timeout.

Processors:

tls

Custom TLS settings can be used to override system defaults.

Processors:

proxy_url

HTTP proxy URL. Empty string disables proxying.

Processors:

disable_http2

Disable HTTP/2 and force HTTP/1.1.

Processors:

tps_limit

Rate limit in requests per second. 0 disables rate limiting.

Processors:

tps_burst

Maximum burst size for rate limiting.

Processors:

backoff

Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active.

Processors:

tcp

TCP socket configuration.

Processors:

http

HTTP transport settings controlling connection pooling, timeouts, and HTTP/2.

Processors:

access_log_level

Log level for HTTP request/response logging. Empty disables logging.

Processors:

access_log_body_limit

Maximum bytes of request/response body to include in logs. 0 to skip body logging.

Processors:

schema_metadata

When set, the processor reads a schema in benthos common schema format from this metadata key on each message, converts it to the format specified by format, registers it with the schema registry under the configured subject, and encodes the message. When empty (the default), the processor pulls the latest schema from the registry instead.

Processors:

format

The encoding format to use when converting a common schema from metadata.

Processors:

normalize

Whether to normalize the schema before registering with the schema registry (schema_metadata mode only).

Processors:

avro

Configuration for Avro encoding.

Processors:

continue

Indicates whether, if this case passes for a message, the next case should also be tested.

Processors:

Deprecations

The following fields are now deprecated:

Field Description Affected components

commit_timeout

Use max_elapsed_time instead.

Outputs:

avro_raw_json

Use avro.raw_json instead.

Processors:

Default value changes

This release includes the following default value changes:

Field Old default New default Description Affected components

target_address

none

The target address to write to.

Outputs:

commit_timeout

60s

Use max_elapsed_time instead.

Outputs:

Removed components

The following deprecated components have been removed:

  • legacy_redpanda_migrator input and output

  • legacy_redpanda_migrator_offsets input and output

  • redpanda_migrator_bundle input and output

Use the unified redpanda_migrator input and output instead.

Version 4.81.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Description

aws_cloudwatch_logs

Input

Consumes log events from AWS CloudWatch Logs.

New field support

This release adds support for the following new fields:

Field Description Affected components

timeout_seconds

The timeout in seconds, for querying the TigerBeetle cluster.

Inputs:

subject

An optional subject to set for messages.

Outputs:

shared_drives

Whether or not to include shared drives.

Processors:

Version 4.80.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Description

iceberg

Output

Write data to Apache Iceberg tables via REST catalog.

Version 4.79.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Description

aws_dynamodb_cdc

Inputs

Consumes records from DynamoDB Streams with automatic checkpointing and shard management.

New field support

This release adds support for the following new fields:

Field Description Affected components

encoding

Encoding format for messages in the batch. Options: 'protobuf' or 'json'.

Inputs:

schema_registry

Optional Schema Registry configuration for adding Schema Registry wire format headers to messages.

Inputs:

message_format

The format to expect incoming messages from the rest of the pipeline.

Outputs:

timestamp_format

The format to parse string values for TIMESTAMP, TIMESTAMP_LTZ and TIMESTAMP_NTZ columns. Should be a layout for time.Parse in Go.

Outputs:

Version 4.78.0

See the full release notes.

New field support

This release adds support for the following new fields:

Field Description Affected components

tcp

TCP listener configuration for the HTTP server.

Inputs:

Version 4.77.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Description

elasticsearch_v9

Outputs

Publish messages into an Elasticsearch index.

Version 4.76.1

See the full release notes.

Version 4.76.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Description

ffi

Processor

Invoke a function within a shared library as a processing step.

Requires a cgo-enabled binary. See the installation guides for details.

New field support

This release adds support for the following new fields:

Field Description Affected components

tags

Global tags added to each metric.

Metrics:

Version 4.75.0

See the full release notes.

New field support

This release adds support for the following new fields:

Field Description Affected components

batching

Optional explicit batching policy for the output. Note that when batches are formed at the input level they can be expanded by this policy, but not contracted.

Outputs:

Version 4.74.0

See the full release notes.

Version 4.73.0

See the full release notes.

New field support

This release adds support for the following new fields:

Field Description Affected components

tcp

TCP listener socket configuration.

Inputs:

ttl

An optional TTL to set for items.

Outputs:

Processors:

schema_registry

Schema registry information to publish schemas for tracing data along with the data.

Tracers:

Version 4.72.0

See the full release notes.

Version 4.71.0

See the full release notes.

Component updates

This release adds the following new components:

Component Type Description

redpanda

Tracers

-

New field support

This release adds support for the following new fields:

Field Description Affected components

aws

AWS IAM authentication configuration for PostgreSQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password.

Inputs:

offset_header

Header name to add to migrated records containing the source offset for exact consumer group migration. If empty, no offset header is added and exact offset translation is disabled.

Outputs:

max_in_flight

Maximum number of batches to have in flight at any given time. For optimal throughput, set this to the total number of partitions being copied in parallel (up to all partitions in the cluster).

Outputs:

Version 4.70.0

See the full release notes.

Bloblang updates

This release adds the following new Bloblang capabilities:

  • Functions:

    • bytes: Create a new byte array that is zero initialized.

  • Methods:

    • repeat: Creates a new array by repeating the elements of the input array a specified number of times.

New field support

This release adds support for the following new fields:

Field Description Affected components

tcp

Configure TCP socket-level settings to optimize network performance and reliability.

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Config:

Caches:

Inputs:

Outputs:

Processors:

Metrics:

regexp_topics_include

A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the metadata_max_age interval.

Inputs:

regexp_topics_exclude

A list of regular expression patterns for excluding topics. Topics matching any of these patterns will be excluded from consumption, even if they match include patterns.

Inputs:

max_parallel_snapshot_tables

The number of tables that will be processed in parallel during the snapshot processing stage.

Inputs:

tls

Custom TLS settings to override system defaults.

Inputs:

persistent

If set to true, the message is marked as persistent, ensuring it is stored durably and not lost if an intermediary (such as a broker) restarts. By default, messages are not durable.

Outputs:

target_capabilities

Lists the extension capabilities the sender desires from the target, such as support for queues, topics, durability, sharing, or temporary destinations.

Outputs:

message_properties_to

The node that is the intended destination of the message, which may differ from the node currently receiving the transfer.

Outputs:

provenance_header

Header name to add to migrated records indicating their source cluster. When set, each migrated message receives a header with this name containing the source cluster’s seed broker addresses, enabling downstream systems to track message origins for auditing, debugging, or multi-cluster orchestration workflows.

Outputs:

Deprecations

The following fields are now deprecated:

Field Description Affected components

regexp_topics

This boolean field is deprecated in favor of regexp_topics_include and regexp_topics_exclude, which provide more explicit control over topic pattern matching. To migrate, replace regexp_topics: true with regexp_topics_include containing your topic patterns, and optionally add regexp_topics_exclude to filter out specific topics. For example, if you had topics: ["events_."] with regexp_topics: true, change to regexp_topics_include: ["events_."].

Inputs:

Default value changes

This release includes the following default value changes:

Field Old default New default Description Affected components

metadata_max_age

5m

1m

The default changed from 5m to 1m to provide more responsive topic and partition discovery. This means metadata refreshes occur more frequently by default. If you experience increased broker load or want to maintain the previous behavior, explicitly set this field to 5m.

Config:

Caches:

Inputs:

Outputs:

Version 4.68.0

See the full release notes.

Component updates

This release adds the following new components:

  • Processors:

    • jira: Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically.

This release adds support for the following new fields:

  • Inputs components

    • redpanda:

      • unordered_processing - Configures partition consumers to allow parallel and therefore unordered processing of messages of any given partition. This allows for better utilization of processing threads and asynchronous publishing at the output level. The maximum parallelization of each partition is determined by the checkpoint_limit field.

Deprecations

The kafka, kafka_franz, and redpanda_common inputs and outputs are now deprecated. Their functionality has been consolidated into the unified redpanda input and output components. Migrate existing configurations to use the new redpanda components for continued support and access to the latest features.

Version 4.67.5

See the full release notes.

Component updates

This release adds the following new components:

  • Inputs:

    • microsoft_sql_server_cdc: Streams change data from a Microsoft SQL Server database into Redpanda Connect using Change Data Capture (CDC).

This release adds support for the following new fields:

  • Inputs components

    • redpanda_migrator

      • schema_registry - Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters.

  • Outputs components

    • redpanda_migrator

      • schema_registry - Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters.

      • consumer_groups - Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed.

      • topic_replication_factor - The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes.

      • sync_topic_acls - Whether to synchronize topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts.

      • serverless - Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by Redpanda Serverless environments.

Deprecations

This release deprecates the individual migrator components and the bundle wrapper in favor of the unified redpanda_migrator architecture.

Deprecated components:

  • redpanda_migrator input and output (renamed to legacy_redpanda_migrator)

  • redpanda_migrator_offsets input and output (renamed to legacy_redpanda_migrator_offsets)

  • redpanda_migrator_bundle input and output (now deprecated; these are part of the legacy migration architecture and internally depend on the deprecated legacy_redpanda_migrator and legacy_redpanda_migrator_offsets components)

These components have been renamed from their previous versions (without the legacy_ prefix) to preserve backward compatibility. The new redpanda_migrator components introduced in this release contain breaking changes. Existing configurations using the previous redpanda_migrator components should use these legacy_ prefixed versions to maintain current functionality.

Users should migrate from deprecated components to the new unified redpanda_migrator input/output pair. For detailed migration instructions, see Migrate to the Unified Redpanda Migrator.

Version 4.66.0

See the full release notes.

Component updates

This release adds the following new components:

  • Outputs:

    • cyborgdb: This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization.

All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure.

  • Bloblang-functions:

    • with_schema_registry_header: Prepends a 5-byte Schema Registry header to a message. The header consists of a magic byte (0x00) followed by a 4-byte big-endian schema ID.

This release adds support for the following new fields:

Version 4.65.0

See the full release notes.

Component updates

This release adds the following new components:

Model Context Protocol (MCP) server

You can now build and run an MCP server directly in Redpanda Connect. The MCP server lets you expose your Redpanda Connect configurations as AI-consumable HTTP endpoints, making them discoverable and callable by AI clients such as Claude Code and other compatible agents.

Version 4.64.0

See the full release notes.

Component updates

This release adds support for the following new fields and improvements:

  • Schema registry decode processor:

    • default_schema_id field for specifying a fallback schema ID when decoding messages without an explicit schema reference.

Version 4.62.0

Component updates

This release adds support for the following new fields and improvements:

  • Kafka, Redpanda, and Franz input components:

  • Redpanda Migrator and Redpanda Common input components:

  • Schema Registry output component:

  • Parquet Encode processor:

  • Socket output component:

  • Bloblang:

    • infer_schema method attempts to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods.

Version 4.61.0

Component updates

This release adds support for the following new fields:

Version 4.60.0

Component updates

The 4.60.0 release adds support for the following fields:

  • NATS components

    • tls_handshake_first - Enhanced TLS configuration option for NATS connections that require the initial TLS handshake to be performed before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation.

This field is available in the following NATS components:

Version 4.59.0

Redpanda Connect in Redpanda Cloud: GA

Redpanda Connect is now generally available (GA) in all Redpanda Cloud clusters: BYOC (including BYOVPC/BYOVNet), Dedicated, and Serverless.

Redpanda Connect has been GA in Redpanda Streaming clusters since version 4.29.0.

Component updates

This release adds support for the following fields:

Version 4.58.2

This release introduces an experimental Slack output and expands support for automatic topic creation across Redpanda-related components.

New components

  • slack_reaction: Adds or removes emoji reactions from Slack messages by channel ID and timestamp.

Component updates

This release adds support for the allow_auto_topic_creation field in the following components:

This field allows components to create topics automatically when publishing data, if the underlying Kafka/Redpanda broker is configured to allow it.

Version 4.57.0

This release adds support for the following fields:

Version 4.56.0

This release introduces a dynamic plugins framework. It enables you to create and load custom plugins at runtime using any programming language that supports gRPC, with initial SDKs for Go and Python.

New components

  • gcp_spanner_cdc: Streams change data from a Google Cloud Spanner instance.

Component updates

This release adds support for the following fields:

  • scope in the couchbase cache.

  • poll_interval and max_yield_batch_bytes in the redpanda_migrator_offsets input.

  • consumer_group_offsets_poll_interval in the redpanda_migrator_bundle input.

  • input_bundle_label in the redpanda_migrator_bundle output.

  • object_canned_acl in the aws_s3 output.

  • history, max_tool_calls, and tools in the gcp_vertex_ai_chat processor.

Version 4.55.1

This release adds support for the following fields:

Version 4.55.0

This release introduces a new beta Redpanda cache component and new fields across several inputs and outputs.

New components

  • redpanda: A new cache that stores key/value pairs in a compacted Redpanda topic.

Component updates

This release adds support for the following fields:

Removed components

The deprecated elasticsearch output has been removed. Use the elasticsearch_v8 output instead.

New Bloblang features

Version 4.54.0 to 4.54.1

These releases introduce the qdrant processor, which allows you to query items within a Qdrant collection and filter the returned results.

Component updates

These releases add:

You can also now use Bloblang to generate a name-based UUID (version 5) for a given string.

Version 4.53.0

This release introduces three new Google Drive processors:

The release also adds the cohere_rerank processor, which sends document strings to the Cohere API, and returns them ranked by their relevance to a specified query.

Component updates

This release adds:

Deprecations

This release deprecates the start_from_oldest field from the following components: kafka_franz input, ockam_kafka input, redpanda input, redpanda_common input, redpanda_migrator input. Now, you must use the start_offset field instead, which defaults to earliest. Redpanda recommends upgrading to benefit from this update.

Version 4.52.0

This release introduces three new Slack components:

Version 4.51.0

Highlights of this update include the introduction of three new components that you could use as part of an AI toolchain:

You can also now install a FIPS-compliant version of Redpanda Connect (for RHEL/Debian Ubuntu).

Component updates

This release adds:

Version 4.50.0

This release introduces agentic behavior to the openai_chat_completion processor. You can now specify external tools to enrich the responses of OpenAI large language models.

Component updates

This release:

Redpanda Connect also now supports the unicode_segments Bloblang method, which splits a string into segments using Unicode text segmentation rules.

Version 4.49.1

This release introduces further updates to the postgres_cdc input and snowflake_streaming output.

Component updates

This release adds:

Also, the translate_schema_ids field in the redpanda_migrator output now defaults to false.

Deprecations

This release deprecates the snapshot_memory_factor field from the postgres_cdc input. Now, you must set a batch size, which defaults to 1000 messages. Redpanda recommends upgrading to benefit from this update.

Version 4.48.1

This release introduces a change data capture (CDC) input for MongoDB on Cloud and Self-Managed Redpanda Connect, along with improvements to the postgres_cdc input and snowflake_streaming output.

Component updates

This release adds:

  • The heartbeat_interval field to the postgres_cdc input, which helps manage the size of the Write-Ahead Log when you subscribe to data changes from tables with low activity.

  • The commit_timeout field to the snowflake_streaming output.

  • The url field to the snowflake_streaming output, which allows you to override the default connection URL.

  • The is_high_watermark field to the redpanda_migrator_offsets output.

  • The kafka_is_high_watermark metadata field to the redpanda_migrator_offsets input.

Other changes:

  • You can now load an Enterprise Edition license using the REDPANDA_LICENSE environment variable.

  • The postgres_cdc input no longer adds the prefix _rs to the replication slots it creates.

  • The clickhouse driver is now supported in Redpanda Cloud for all components with the prefix sql_.

  • The redpanda migrator output now attempts to transfer access control lists for topics that already exist on the destination broker.

Version 4.47.0 to 4.47.1

This release introduces:

  • The elasticsearch_v8 output: A certified, up-to-date alternative to the existing elasticsearch output.

  • The crash processor: A tool for detecting unhandled pipeline errors during development and testing.

Component updates

This release adds:

Redpanda Connect also now supports the uuid_v7 Bloblang function, which is useful for scenarios when the exact timing of events is important, such as during data migration or replication.

Version 4.46.0

This release introduces a change data capture (CDC) input for MySQL, and enhancements to the Bloblang playground, making it easier to test and debug your pipeline configurations. The enhancements include:

  • A Run button in Bloblang code snippets, which opens each snippet in the playground with a pre-filled input and mapping ready for testing.

  • A Get help from AI button, which appears in the playground when an error occurs.

Components for Cloud and Self-Managed Redpanda Connect (beta)

Component updates

This release adds:

Other changes:

Deprecations

All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect.

Deprecated fields Affected components

new_column_type_mapping

snowflake_streaming output

Version 4.45.0

Highlights of this update include the beta release of two components for Self-Managed and Cloud Redpanda Connect, and a new cookbook that explains how to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. You can also now share links to code snippets in the Bloblang playground using the Share link button.

Components for Cloud and Self-Managed Redpanda Connect (beta)

  • ollama_moderation processor (Self-Managed, Cloud)

  • redpanda_migrator_offsets input (Self-Managed, Cloud)

Component updates

This release adds:

Redpanda Connect now supports:

Deprecations

All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect.

Deprecated fields Affected components

batching

redpanda_migrator input, redpanda_migrator output

kafka_key, max_in_flight

redpanda_migrator_offsets output

batch_size, multi_header, replication_factor, replication_factor_override, output_resource

redpanda_migrator input

Version 4.44.0

This release extends the functionality of the openai_chat_completion processor to include the following fields: max_tokens, temperature, user, top_p, frequency_penalty, presence_penalty, seed, and stop.

Version 4.43.1

This release introduces support for trial licenses for Self-Managed Redpanda Connect.

Version 4.43.0

This release renames the pg_stream input to postgres_cdc.

The postgres_cdc input also no longer emits mode metadata, but sets the operation metadata field to read when a snapshot of a database is processed.

Version 4.42.0

This release introduces a Bloblang playground where you can try out your Bloblang syntax and get immediate feedback. It also updates Redpanda Connect licensing checks for Redpanda Connect. Starting from version 4.42.0, you must add your Enterprise Edition license to Redpanda Connect before using any Enterprise connectors.

Component updates

Redpanda Connect now supports:

  • A Google Cloud spanner driver for all SQL plugins.

  • The following complex data types for the pg_stream input: JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[].

  • Parquet files for the bigquery output.

This release also adds an exists operator to the cache processor.

Version 4.41.0

This release adds the max_records_per_request field to the aws_sqs output.

Packaging update

Starting from version 4.41.0, the -cgo suffixed Docker image is no longer available due to low demand and the unacceptable cadence with which the image base (Debian) receives security updates.

To create your own CGO builds, run the following command:

CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect.

Version 4.40.0

This update includes a beta release of:

  • The pg_stream input, which captures data changes made to a PostgreSQL database in real time and streams them to Redpanda Connect.

  • The benchmark processor, which provides message throughput statistics for a data pipeline.

It also introduces a new Bloblang method for timestamps.

Components for Cloud and Self-Managed Redpanda Connect (beta)

Component updates

This release adds:

Other changes:

Deprecations

Deprecated field: kafka_timestamp metadata field

Affected components:

Replaced by: kafka_timestamp_ms metadata field

Deprecated field: timestamp

Affected components:

Replaced by: timestamp_ms field

Action required: Modify all systems that process these fields to ensure compatibility with the new field names.

Version 4.39.0

Highlights of this update include the beta release of three new components for Self-Managed and Cloud, including a new snowflake_streaming output. This release also includes additions to the redpanda family of components and introduces new features for Self-Managed.

Components for Cloud and Self-Managed (beta)

Additional redpanda components (beta)

  • redpanda (input and output): Consume message data from or send message data to Kafka brokers.

  • redpanda_common (input and output): Consume message data from or send message data to a Redpanda Kafka broker, using credentials from a common redpanda configuration block (Self-Managed only).

Features in Self-Managed

You can now:

Component updates

This release adds:

Deprecations

All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect.

Deprecated fields Affected components

auto_replay_nacks, sasl, metadata_max_age

ockam_kafka input

client_id, rack_id, sasl

ockam_kafka output

rack_id

kafka_franz output

Version 4.38.0

This update includes the beta release of two new outputs for Cloud and Self-Managed, and the introduction of a telemetry service, which collects and sends anonymized usage statistics to Redpanda.

New tooling

A new Helm chart for deploying Redpanda Connect in Kubernetes is now available. This chart simplifies the deployment and configuration of Redpanda Connect, allowing you to manage and scale Redpanda Connect instances in Kubernetes environments.

To get started with the Helm chart, see Get Started with the Redpanda Connect Helm Chart.

Or, see the release notes for the chart on GitHub.

New outputs (beta)

Component updates

This release adds:

Other changes:

  • The collection field of the mongodb output now supports interpolation functions.

  • All components that used the deprecated count function, with a default value for the path field, now use the counter function.

    If you have multiple components that execute a mapping using the deprecated count function with the original default value, and these counters need to cascade, there may be a change in behavior. To avoid this, explicitly define a value for the path field.

Version 4.37.0

Redpanda Connect has new bundled Redpanda Migrator components, more AI processors for Self-Managed and Cloud platforms, and three new outputs.

Redpanda Migrator components (beta)

You can now move workloads from any Apache Kafka system to Redpanda using the redpanda_migrator_bundle input and output.

For full details, see the Redpanda Migrator Cookbook for Self-Managed and Cloud.

AI processors (beta)

Some new additions to Redpanda Connect’s growing list of AI processors:

Certified and community outputs (beta)

Component updates

This release adds:

Version 4.36.0

You can now use the replication_factor_override and replication_factor fields of the redpanda_migrator input and output to set a replication factor for migrated topics.

Versions 4.35.1 to 4.35.3

Redpanda Connect is now fully integrated with the Redpanda Cloud platform. This release adds Azure and GCP components to the growing list of available components. For a full list of components supported in Redpanda Cloud, see the Connector Catalog.

Version 4.35.0

Redpanda Connect now includes beta versions of Redpanda Migrator components:

For full details, see the Redpanda Migrator Cookbook for Self-Managed and Cloud.

Component updates

This release adds:

Version 4.34.0

Redpanda Connect now includes beta versions of two new AI processors for Cloud and Self-Managed platforms:

Version 4.33.0

Redpanda Connect now includes beta versions of four new components.

For Self-Managed and Cloud platforms

For Self-Managed only

You can also use the new Bloblang vector method to create vector embeddings for SQL databases.

Component updates

This release adds:

  • The credentials_json field to all GCP components.

  • The content_md5 field to the aws_s3 output.

  • The send_ack field to the nats input.

Version 4.32.1

Redpanda Connect has a new batch of beta AI processors for Self-Managed and Cloud platforms.

For Self-Managed and Cloud platforms

For Self-Managed only

Version 4.31.0

Redpanda Connect now includes beta versions of two new components:

Component updates

This release adds:

Version 4.30.0 to 4.30.1

Redpanda Connect now includes a beta version of the splunk input.

Version 4.29.0

Redpanda Connect now separates packages into an enterprise bundle and a free bundle, which contains community and selected certified connectors. To learn more about licensing and support levels, see components:about.adoc#support-levels.

Component updates

This release adds: