Migrate Data with Redpanda Migrator

This lab demonstrates how to use Redpanda Migrator to migrate data from a source Kafka-compatible cluster to a target Redpanda cluster.

Redpanda Migrator is built on Redpanda Connect and provides continuous data replication with automatic topic creation, schema migration, and consumer offset synchronization.

What you’ll explore

In this lab, you will:

  • Run Redpanda Migrator for continuous data replication between clusters

  • Explore SASL authentication and ACL-based authorization for secure migration

  • Observe topic and schema migration with preserved configurations

Architecture

This lab creates a complete migration environment:

  • Source cluster: Simulates a Confluent Kafka cluster with SASL authentication, demo topics, and registered schemas

  • Target cluster: Redpanda cluster with matching security configuration

  • Redpanda Migrator: Continuously replicates data, schemas, and offsets from source to target

  • Console instances: Web UIs for both clusters to observe the migration in real-time

The migrator automatically creates topics with matching partition counts and configurations, preserves schema IDs and versions, and migrates consumer offsets.

Prerequisites

This lab is for Linux and macOS users. If you are using Windows, you must use the Windows Subsystem for Linux (WSL) to run the commands in this lab.

Run the lab

  1. Clone this repository:

    git clone https://github.com/redpanda-data/redpanda-labs.git

  2. Change into the docker-compose/redpanda-migrator-demo/ directory:

    cd redpanda-labs/docker-compose/redpanda-migrator-demo

  3. (Optional) Set container versions.

    The docker-compose.yml file uses default versions. To use specific versions, set these environment variables:

    export REDPANDA_VERSION=25.3.4
export REDPANDA_CONSOLE_VERSION=3.4.0
export REDPANDA_CONNECT_VERSION=4.77.0

    For all available Redpanda versions, see the GitHub releases.

  4. Start the demo environment:

    make start

    This command starts the source and target Redpanda clusters, Redpanda Console instances, and the migrator service. See docker-compose.yml for port mappings and service details.

  5. Set up users, ACLs, topics, and schemas:

    make setup

    This creates users with SASL authentication, configures ACLs for secure migration, creates demo topics with various cleanup policies, and registers schemas. See scripts/setup.sh for detailed ACL permissions and topic configurations.

  6. (Optional) Verify the ACL configuration:

    make verify-acls

    This confirms that migrator-user has correct read-only source access and read-write target access with proper restrictions. See scripts/verify-acls.sh for test details.

  7. Start continuous data production:

    make demo-start

    This starts a background producer that sends approximately 80 messages every 2 seconds across all demo topics.

  8. (Optional) View the migration in Redpanda Console:

    Open your browser to observe the migration in real-time:

  9. (Optional) Verify continuous schema syncing:

    make verify-continuous-schema

    This command registers a new schema after the migrator starts and confirms it automatically appears in the target Schema Registry within 15 seconds.

  10. Monitor migration lag:

    make monitor-lag

    This command displays a real-time dashboard showing lag per topic, message counts, and production status. Watch as the lag decreases to 0, indicating the migrator has caught up with the source cluster. Let it run for 30-60 seconds to observe continuous replication, then press Ctrl+C to exit.

  11. Verify data consistency:

    Run the verification script to confirm successful migration:

    make verify

    This command checks that all topics, partitions, messages, cleanup policies, and schemas have been migrated correctly.

  12. Stop continuous production:

    make demo-stop

  13. Disable Schema Registry import mode:

    This step is required to return the target Schema Registry to normal operations.

    The target Schema Registry must be in import mode for schema migration to work (enabled automatically by make setup). After migration completes, disable import mode:

    curl -X PUT http://localhost:28081/mode \
  -H "Content-Type: application/json" \
  -u 'admin-user:admin-secret-password' \
  -d '{"mode":"READWRITE"}'

    Import mode allows the migrator to register schemas with preserved IDs and versions. When migration is done, return to readwrite mode for normal operations.

Configuration

The migrator uses the principle of least privilege with non-superuser accounts. For detailed security configuration including users, ACLs, and permissions, see:

  • scripts/setup.sh: User creation and ACL configuration

  • config/migrator-config.yaml: Migrator settings and Schema Registry sync

  • docker-compose.yml: Service ports and environment variables

This lab uses single-node clusters and demo credentials for simplicity. For production migrations, use multi-node clusters with strong, randomly generated passwords stored in a secrets manager such as HashiCorp Vault or AWS Secrets Manager.

Troubleshoot

Authentication errors

If you see "ILLEGAL_SASL_STATE" errors, run make setup to configure SASL authentication.

Topics not appearing in target

Check that the migrator is running:

docker compose ps migrator
curl http://localhost:4195/ping

Check the migrator logs:

docker compose logs migrator

Schemas not migrating

If you see "Subject is not in import mode" errors, enable import mode:

curl -X PUT http://localhost:28081/mode \
  -H "Content-Type: application/json" \
  -u 'admin-user:admin-secret-password' \
  -d '{"mode":"IMPORT"}'

Then restart the migrator:

docker compose restart migrator

Clean up

To stop and remove all containers and data:

make clean

Next steps

After running this demo: