# Migrate Data with Redpanda Migrator

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [labs-full.txt](https://docs.redpanda.com/labs-full.txt)

---
title: Migrate Data with Redpanda Migrator
latest-operator-version: v26.1.4
latest-console-tag: v3.7.3
latest-connect-version: 4.93.0
latest-redpanda-tag: v26.1.9
docname: redpanda-migrator
page-component-name: labs
page-version: master
page-component-version: master
page-component-title: Labs
page-relative-src-path: redpanda-migrator.adoc
page-edit-url: https://github.com/redpanda-data/redpanda-labs/edit/main/docs/modules/docker-compose/pages/redpanda-migrator.adoc
description: Migrate data, schemas, and consumer offsets from a source Kafka cluster to a target Redpanda cluster using Redpanda Migrator.
page-topic-type: lab
personas: streaming_developer, platform_operator, evaluator
learning-objective-1: Run Redpanda Migrator for continuous data replication between clusters
learning-objective-2: Explore SASL authentication and ACL-based authorization for secure migration
learning-objective-3: Observe topic and schema migration with preserved configurations
page-git-created-date: "2026-01-12"
page-git-modified-date: "2026-01-12"
---

<!-- Source: https://docs.redpanda.com/labs/docker-compose/redpanda-migrator.md -->

This lab demonstrates how to use [Redpanda Migrator](https://docs.redpanda.com/connect/components/inputs/redpanda_migrator/) to migrate data from a source Kafka-compatible cluster to a target Redpanda cluster.

Redpanda Migrator is built on Redpanda Connect and provides continuous data replication with automatic topic creation, schema migration, and consumer offset synchronization.

## [](#what-youll-explore)What you’ll explore

In this lab, you will:

-   Run Redpanda Migrator for continuous data replication between clusters

-   Explore SASL authentication and ACL-based authorization for secure migration

-   Observe topic and schema migration with preserved configurations


## [](#architecture)Architecture

This lab creates a complete migration environment:

-   **Source cluster**: Simulates a Confluent Kafka cluster with SASL authentication, demo topics, and registered schemas

-   **Target cluster**: Redpanda cluster with matching security configuration

-   **Redpanda Migrator**: Continuously replicates data, schemas, and offsets from source to target

-   **Console instances**: Web UIs for both clusters to observe the migration in real-time


The migrator automatically creates topics with matching partition counts and configurations, preserves schema IDs and versions, and migrates consumer offsets.

## [](#prerequisites)Prerequisites

-   [Docker and Docker Compose](https://docs.docker.com/compose/install/)

-   At least 4GB RAM available

-   [GNU Make](https://www.gnu.org/software/make/) (usually pre-installed on Linux/macOS)


This lab is for Linux and macOS users. If you are using Windows, you must use the Windows Subsystem for Linux (WSL) to run the commands in this lab.

## [](#run-the-lab)Run the lab

1.  Clone this repository:

    ```bash
    git clone https://github.com/redpanda-data/redpanda-labs.git
    ```

2.  Change into the `docker-compose/redpanda-migrator-demo/` directory:

    ```bash
    cd redpanda-labs/docker-compose/redpanda-migrator-demo
    ```

3.  (Optional) Set container versions.

    The `docker-compose.yml` file uses default versions. To use specific versions, set these environment variables:

    ```bash
    export REDPANDA_VERSION=26.1.9
    export REDPANDA_CONSOLE_VERSION=3.7.3
    export REDPANDA_CONNECT_VERSION=4.93.0
    ```

    For all available Redpanda versions, see the [GitHub releases](https://github.com/redpanda-data/redpanda/releases).

4.  Start the demo environment:

    ```bash
    make start
    ```

    This command starts the source and target Redpanda clusters, Redpanda Console instances, and the migrator service. See `docker-compose.yml` for port mappings and service details.

5.  Set up users, ACLs, topics, and schemas:

    ```bash
    make setup
    ```

    This creates users with SASL authentication, configures ACLs for secure migration, creates demo topics with various cleanup policies, and registers schemas. See `scripts/setup.sh` for detailed ACL permissions and topic configurations.

6.  (Optional) Verify the ACL configuration:

    ```bash
    make verify-acls
    ```

    This confirms that `migrator-user` has correct read-only source access and read-write target access with proper restrictions. See `scripts/verify-acls.sh` for test details.

7.  Start continuous data production:

    ```bash
    make demo-start
    ```

    This starts a background producer that sends approximately 80 messages every 2 seconds across all demo topics.

8.  (Optional) View the migration in Redpanda Console:

    Open your browser to observe the migration in real-time:

    -   Source cluster: [http://localhost:8080](http://localhost:8080)

    -   Target cluster: [http://localhost:8081](http://localhost:8081)

        You can see topics, messages, and schemas being replicated from source to target.


9.  (Optional) Verify continuous schema syncing:

    ```bash
    make verify-continuous-schema
    ```

    This command registers a new schema after the migrator starts and confirms it automatically appears in the target Schema Registry within 15 seconds.

10.  Monitor migration lag:

     ```bash
     make monitor-lag
     ```

     This command displays a real-time dashboard showing lag per topic, message counts, and production status. Watch as the lag decreases to 0, indicating the migrator has caught up with the source cluster. Let it run for 30-60 seconds to observe continuous replication, then press Ctrl+C to exit.

11.  Verify data consistency:

     Run the verification script to confirm successful migration:

     ```bash
     make verify
     ```

     This command checks that all topics, partitions, messages, cleanup policies, and schemas have been migrated correctly.

12.  Stop continuous production:

     ```bash
     make demo-stop
     ```

13.  Disable Schema Registry import mode:

     > ❗ **IMPORTANT**
     >
     > This step is required to return the target Schema Registry to normal operations.

     The target Schema Registry must be in import mode for schema migration to work (enabled automatically by `make setup`). After migration completes, disable import mode:

     ```bash
     curl -X PUT http://localhost:28081/mode \
       -H "Content-Type: application/json" \
       -u 'admin-user:admin-secret-password' \
       -d '{"mode":"READWRITE"}'
     ```

     Import mode allows the migrator to register schemas with preserved IDs and versions. When migration is done, return to readwrite mode for normal operations.


## [](#configuration)Configuration

The migrator uses the principle of least privilege with non-superuser accounts. For detailed security configuration including users, ACLs, and permissions, see:

-   `scripts/setup.sh`: User creation and ACL configuration

-   `config/migrator-config.yaml`: Migrator settings and Schema Registry sync

-   `docker-compose.yml`: Service ports and environment variables


This lab uses single-node clusters and demo credentials for simplicity. For production migrations, use multi-node clusters with strong, randomly generated passwords stored in a secrets manager such as HashiCorp Vault or AWS Secrets Manager.

## [](#troubleshoot)Troubleshoot

### [](#authentication-errors)Authentication errors

If you see "ILLEGAL\_SASL\_STATE" errors, run `make setup` to configure SASL authentication.

### [](#topics-not-appearing-in-target)Topics not appearing in target

Check that the migrator is running:

```bash
docker compose ps migrator
curl http://localhost:4195/ping
```

Check the migrator logs:

```bash
docker compose logs migrator
```

### [](#schemas-not-migrating)Schemas not migrating

If you see "Subject is not in import mode" errors, enable import mode:

```bash
curl -X PUT http://localhost:28081/mode \
  -H "Content-Type: application/json" \
  -u 'admin-user:admin-secret-password' \
  -d '{"mode":"IMPORT"}'
```

Then restart the migrator:

```bash
docker compose restart migrator
```

## [](#clean-up)Clean up

To stop and remove all containers and data:

```bash
make clean
```

## [](#next-steps)Next steps

After running this demo:

-   Read the [Redpanda Migrator documentation](https://docs.redpanda.com/connect/cookbooks/redpanda_migrator/)

-   Learn about [Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/) configuration

-   Explore [Redpanda authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/) options