Migrate to the Unified Redpanda Migrator
| This page is about migrating to a newer version of Redpanda Connect. For information about migrating your data using Redpanda Migrator, see Redpanda Migrator. |
This guide explains how to migrate from legacy migrator components (redpanda_migrator_bundle, legacy_redpanda_migrator and legacy_redpanda_migrator_offsets) to the unified redpanda_migrator input/output pair introduced in Redpanda Connect 4.67.5+.
The unified migrator consolidates all migration logic into a single input/output pair, simplifying configuration and improving reliability.
Overview
Available in |
Redpanda Connect 4.67.5+ |
Legacy status |
Deprecated in 4.67.5, removed in 4.85.0 |
Compatibility |
Not backward-compatible |
Configuration model |
One |
Primary control |
All migration logic resides in the output component |
Key concepts:
-
Components are paired by matching
labelvalues. -
The input defines the source cluster and schema registry.
-
The output defines the destination cluster, schema registry, and migration behavior.
-
Topic mapping and consumer group migration are configured in the output.
Architectural changes
Legacy architecture
A complex bundle (redpanda_migrator_bundle) that managed three subcomponents:
-
redpanda_migrator: Data transfer -
schema_registry: Schema synchronization -
redpanda_migrator_offsets: Consumer group offsets
This design required complex internal routing and sequencing.
Unified architecture
A single redpanda_migrator input/output pair replaces the bundle:
-
Input: Consumes from the source Kafka cluster.
-
Output: Handles topic creation, schema synchronization, ACLs, and consumer group offsets.
Benefits:
-
Simplified setup: all configuration consolidated in one output component.
-
Improved coordination: no internal routing or wrapper logic.
-
Enhanced control: fine-grained schema and topic options, improved offset handling.
Migration steps
Follow this checklist in order to ensure a safe, low-risk migration.
-
Back up your existing configurations.
-
Add new
input.redpanda_migratorandoutput.redpanda_migratorcomponents with matching labels. -
Move source Kafka and Schema Registry settings to the input.
-
Move destination Kafka and Schema Registry settings to the output.
-
Replace
topic_prefixwithtopicusing interpolation syntax. -
Move offset settings to
output.redpanda_migrator.consumer_groups. -
Remove deprecated fields.
-
Validate configuration with
rpk connect lint. -
Test using non-production topics first.
-
Monitor logs and performance during migration.
-
Remove legacy configuration after successful migration.
Field mapping reference
Bundle wrapper (redpanda_migrator_bundle)
Input mapping
| Legacy Field | New Location | Status | Notes |
|---|---|---|---|
|
|
Moved |
Source cluster connection |
|
|
Moved |
Source schema registry |
|
- |
Removed |
Controlled by output schema interval |
|
|
Moved |
Now controls sync frequency |
Output mapping
| Legacy Field | New Location | Status | Notes |
|---|---|---|---|
|
|
Moved |
Destination cluster configuration |
|
|
Moved |
Destination schema registry |
|
|
Moved |
Schema ID translation |
|
|
Replaced |
Input and output paired by label |
Data migration fields
| Legacy Field | New Location | Status | Notes |
|---|---|---|---|
All ( |
|
Moved |
Direct mapping |
|
|
Unchanged |
Still supported for explicit lists |
|
|
Deprecated |
Use include/exclude arrays for pattern-based selection |
|
|
Replaced |
Use interpolation, for example |
|
|
Replaced |
Unified field |
|
|
Replaced |
Label pairing replaces internal routing |
- |
|
New |
Optional header for tracking message source cluster |
Schema migration fields
| Legacy Field | New Location | Status | Notes |
|---|---|---|---|
Connection fields |
|
Moved |
Source schema registry |
|
|
Replaced |
Use regex lists for filtering |
|
|
Moved |
Configured on destination |
|
|
Replaced |
Choose |
Migration example
The following example demonstrates a complete migration from legacy to unified components.
input:
label: "source_cluster"
redpanda_migrator_bundle:
legacy_redpanda_migrator:
seed_brokers: [ "source-kafka:9092" ]
topics: [ "orders", "payments" ]
consumer_group: "migration_group"
schema_registry:
url: "http://source-registry:8081"
migrate_schemas_before_data: false
consumer_group_offsets_poll_interval: 30s
output:
redpanda_migrator_bundle:
legacy_redpanda_migrator:
seed_brokers: [ "destination-redpanda:9092" ]
topic_prefix: "migrated_"
schema_registry:
url: "http://destination-registry:8081"
translate_schema_ids: true
input_bundle_label: "source_cluster"
input:
label: "migration_pipeline" (1)
redpanda_migrator:
# Source Kafka settings
seed_brokers: [ "source-kafka:9092" ]
# Pattern-based topic selection (for migrating all topics except system topics)
# Note: You can still use explicit lists: topics: [ "orders", "payments" ]
regexp_topics_include: [ '.' ] (2)
regexp_topics_exclude: [ '^_' ] (3)
consumer_group: "migration_group"
# Source Schema Registry settings
schema_registry:
url: "http://source-registry:8081"
output:
label: "migration_pipeline" (4)
redpanda_migrator:
# Destination Redpanda settings
seed_brokers: [ "destination-redpanda:9092" ]
# Topic mapping (replaces topic_prefix)
topic: 'migrated_${! @kafka_topic }' (5)
# Add source cluster tracking header
provenance_header: "x-source-cluster" (6)
# Destination Schema Registry and migration settings
schema_registry:
url: "http://destination-registry:8081"
translate_ids: true
# Rename subjects
subject: 'migrated_${! metadata("schema_registry_subject") }'
# Consumer group migration settings
consumer_groups:
enabled: true
interval: 30s (7)
| 1 | Labels are now used for pairing input and output. |
| 2 | Match all topics using regex pattern. |
| 3 | Exclude internal/system topics starting with underscore. |
| 4 | Matching label pairs the input and output components. |
| 5 | Use interpolation syntax to replicate topic_prefix behavior. |
| 6 | Adds a header to track which cluster messages originated from, useful for debugging and auditing. |
| 7 | Replaces consumer_group_offsets_poll_interval. |
Validation
Before running, validate your configuration:
rpk connect lint config.yaml
Then test on a small set of topics before running full migrations.
Troubleshooting
| Problem | Likely Cause | Solution |
|---|---|---|
Labels do not match |
Input and output labels differ |
Use identical, case-sensitive labels. |
Topic interpolation errors |
Incorrect syntax |
Use |
Schema registry connection fails |
Incorrect registry placement |
The source registry must be in the input. The destination registry must be in the output. |
Consumer group migration not working |
Missing |
Ensure consumer group migration is explicitly enabled. |
After migration
After verifying that the new migrator works as expected:
-
Remove legacy configuration files.
-
Update internal documentation and runbooks.
-
Train your team on the new configuration model.
-
See the
redpanda_migratoroutput reference for advanced configuration options.