Streaming

Migrate from Kafka to Redpanda

Move a running Kafka workload to Redpanda with little downtime using Shadowing. Use Redpanda Migrator only when your migration needs something Shadowing cannot do.

After reading this page, you will be able to:

  • Evaluate whether Shadowing can migrate your Kafka workload

  • Recognize the cases where Redpanda Migrator fits better than Shadowing

  • Apply the cutover check that each path needs before you move a producer

How Shadowing migrates a workload

A shadow link runs inside the Redpanda cluster you migrate to, which acts as the shadow cluster. The cluster connects to the source as a client, pulls the topics that the link’s filters select, and writes each record at the offset it has on the source. Consumers therefore resume on Redpanda at the same offsets, with no translation.

The same link copies consumer group offsets, capped at the target partition’s high watermark, and replicates the ACLs that match its filters. When you configure Schema Registry sync, it also replicates schemas with their IDs. Shadow topics refuse writes until you fail them over, so nothing writes to the target by mistake before the cutover. There is nothing to deploy between the clusters. For details, see Configure Shadowing.

Shadowing has the following requirements:

  • The cluster you migrate to runs Redpanda version 25.3 or later and has an Enterprise Edition license.

  • To replicate schemas from a Confluent Schema Registry, the cluster you migrate to runs version 26.2 or later.

  • The shadow link authenticates to the source with SASL/SCRAM, SASL/PLAIN (version 25.3.2 or later), TLS, or mutual TLS, or connects without authentication.

  • The source Schema Registry is Confluent-compatible and accepts HTTP basic authentication, mutual TLS, or no authentication.

  • An Amazon MSK source has SASL/SCRAM-SHA-512 enabled.

After the migration, you can create a new shadow link that uses the new cluster as its source, to keep a disaster recovery standby.

Cut over with Shadowing

With Shadowing, you move consumers first, while producers still write to the source, and producers last.

  1. Stop each consumer group on the source and wait until the target holds the same committed offset on every partition. Before you start the group on the target, exclude it from the link’s consumer offset sync with an exclude group filter, or pause the consumer offset sync. Otherwise, whenever the group has no active members on the target, the link copies the source’s stopped offsets over the moved consumers' progress.

  2. Pause or stop the producers of the topics you move, then wait until each partition’s high watermark, read from the source cluster, equals the target’s. Compare both clusters directly. The lag that the link reports comes from its last fetch from the source, so it can read 0 while the source holds records the link has not fetched.

    A failover stops replication for good. Any record that is still only on the source when you fail over stays there.

  3. If the moved producers register schemas, pause the link’s Schema Registry sync by setting paused: true in its schema_registry_sync_options. The link’s Schema Registry contexts on the target refuse new schemas for as long as the sync runs, even after the topics fail over. Pausing stops all schema replication from the source registry.

  4. Fail over the topics, then start the producers on Redpanda.

  5. After the last topic fails over, delete the link.

For details, see Failover.

When to use Redpanda Migrator instead

Redpanda Migrator is a Redpanda Connect pipeline that runs between the two clusters and copies records, topic settings, schemas, and consumer group offsets. Use it only when one of these applies:

  • The cluster you migrate to cannot run Shadowing. For example, a cluster that runs a version before 25.3, or has no Enterprise Edition license. Redpanda Migrator needs no Enterprise license.

  • Your source accepts only IAM or OAUTHBEARER authentication. A shadow link has no AWS IAM or OAUTHBEARER support, so Amazon MSK with IAM access control is out of reach. Redpanda Migrator supports both. Neither Shadowing nor Redpanda Migrator reads schemas from AWS Glue Schema Registry.

  • Your rollback plan needs records written to Redpanda after the cutover copied back to Kafka. A shadow link replicates only into a Redpanda cluster, and a failed-over topic cannot return to shadow mode. Redpanda Migrator can also write from Redpanda to Kafka.

  • Records need to change on the way. A shadow link copies records as they are. Redpanda Migrator can rename topics and filter or transform record values with Redpanda Connect processors. Keys, partitions, and timestamps stay as they were on the source.

If none of these applies, use Shadowing.

Redpanda Migrator works differently from Shadowing in ways that change the cutover:

  • Records get new offsets on the target, so the migrator translates each consumer group’s committed offset to the matching record by timestamp. For a group with no active members, it refines that match to the exact record, using a header that holds each record’s source offset. The offset_header field, available in Redpanda Connect 4.71.0 and later, names this header and enables exact translation by default.

  • It copies schemas and, by default, keeps their IDs, which requires the target Schema Registry to be in IMPORT mode. If you set translate_ids, the target assigns new IDs instead.

To cut over with Redpanda Migrator:

  1. Stop each consumer group on the source, so the group has no active members and the migrator translates its offsets to the exact records. Wait for the migrator’s next consumer group sync, check that the translated offsets on the target point at the same records as the source offsets, then start the group on the target. Compare the records the offsets point at, not the offset numbers, which differ between the clusters.

  2. Pause the producers of the topics you move, wait until the migrator’s redpanda_lag metric for those topics is 0, then point the producers at Redpanda. If you move a producer before the lag reaches 0, a record the migrator has not copied yet arrives on the target after newer records with the same key.

  3. After the last group and producer have moved, stop the migrator. If you set the target Schema Registry to IMPORT mode, return it to READWRITE mode.

For details, see Redpanda Migrator.

Next steps