Cloud

Migrate Schemas from Confluent Schema Registry

When you migrate to Redpanda from a deployment that uses a Confluent Schema Registry, your producers and consumers depend on the schemas stored in that registry. Shadowing removes this migration obstacle: a shadow link continuously replicates schemas from the source Confluent Schema Registry into the Schema Registry built into the Redpanda shadow cluster, preserving subject names, versions, and compatibility settings. Because both registries stay synchronized until cutover, your applications keep working on Redpanda without a separate schema migration step. Use this approach when you migrate from Confluent to Redpanda, or when you maintain a Redpanda disaster recovery cluster for a system that keeps its schemas in a Confluent Schema Registry.

After reading this page, you will be able to:

  • Configure a shadow link that continuously replicates schemas from a Confluent Schema Registry

  • Filter replication by context or subject and map source contexts to destination contexts

  • Monitor schema replication status and resolve validation errors

How HTTP API schema replication works

When you configure a shadow link with the shadow_schema_registry_api option, the shadow cluster polls the source Schema Registry over HTTP and imports changes into its own Schema Registry. Two sync cycles keep the registries in step:

  • Tail syncs run frequently (default: every 10 seconds) to pick up incremental changes.

  • Full syncs scan all selected subjects (default: every 5 minutes) to catch anything a tail sync missed.

Replicated schemas keep their original subject names and version IDs, so producers and consumers that reference schemas by ID continue to work after failover. Schemas that reference other schemas import in dependency order.

Before importing a schema, Redpanda validates it against the Redpanda Schema Registry implementation. If a schema uses features that Redpanda does not support, the sync either reports an error and skips the schema, or removes the unsupported fields and imports the rest, depending on the validation policy you choose.

While the link is active, the destination contexts that the link replicates into are read-only: the shadow cluster rejects client writes to those contexts so that replicated schemas remain identical to the source. Contexts outside the link’s filter remain writable.

This API-based mode is an alternative to the byte-for-byte _schemas topic replication described in Configure Shadowing. A shadow link uses one Schema Registry sync mode or the other, not both:

  • Use topic mode (shadow_schema_registry_topic) when the source is another Redpanda cluster and you want an exact, complete replica of its Schema Registry. Topic mode shadows the _schemas topic byte for byte, so it does not filter, remap, or validate schemas.

  • Use API mode (shadow_schema_registry_api) when the source is a Confluent Schema Registry, or when you need to replicate only selected contexts or subjects, map source contexts to different destination contexts, or control how schemas that use unsupported features are handled with a validation policy.

Schema replication settings live in the shadow link configuration. Two cluster properties, schema_registry_sync_memory_bytes and schema_registry_sync_parallelism, tune how much memory and concurrency the shadow cluster uses while importing schemas. The defaults suit most deployments.

Use cases

  • Migrate from Confluent to Redpanda: Replicate schemas continuously while Shadowing replicates your topic data, then cut applications over to Redpanda once both are in sync. No separate schema migration tooling is required.

  • Phased migration: Use context and subject filters to migrate one team, application, or environment at a time.

  • Registry reorganization: Map source contexts to different destination contexts to restructure your Schema Registry as part of the migration.

Prerequisites

  • A cluster running Redpanda version 26.2 or later. The schema replication feature activates after all brokers complete the upgrade.

  • A BYOC or Dedicated cluster. API-mode Schema Registry replication is rolling out to Redpanda Cloud; if the Schema Registry options described on this page are not yet available for your cluster, contact Redpanda Support.

  • Network connectivity from the shadow cluster to the source Schema Registry HTTP endpoint.

  • Credentials for the source registry with permission to read subjects, versions, and configuration. For Confluent Cloud, use a Schema Registry API key and secret.

  • Basic Shadow link settings. See Configure Shadowing.

  • The destination contexts that the link replicates into, as determined by your source_filter and destination mapping, must be empty on the shadow cluster. The rest of the shadow cluster’s Schema Registry does not need to be empty: contexts outside the link’s mappings are unaffected.

  • To replicate contexts other than the default context, the schema_registry_enable_qualified_subjects cluster property must be enabled on the shadow cluster (the default). See Schema Registry Contexts.

Limitations

  • HTTP basic authentication and mTLS are the supported authentication methods for the source registry. You can also connect to a source registry that requires no authentication.

  • Replication is one way, from the source registry to the shadow cluster. Destination contexts owned by the link are read-only until failover.

  • Schemas that use Confluent features not supported by the Redpanda Schema Registry are not replicated as-is. Choose a validation policy to control whether these schemas are skipped or imported without the unsupported fields.

  • Topic data replication and schema replication are coordinated, but not immediately consistent: schema arrival on the shadow cluster can be delayed by the tail refresh interval (tail_interval), which defaults to 10 seconds. Records serialized in the Confluent SerDes wire format can therefore arrive before the schema IDs they contain. This does not cause replication errors, because schema IDs are not validated during topic data replication, but consumers that look up those schema IDs on the shadow cluster fail until the schemas arrive. See Monitor replication status.

  • Role synchronization requires a Redpanda source. Leave role_sync_options unconfigured when the source is a Confluent cluster: a Roles Migrator task pointed at a non-Redpanda source reports LINK_UNAVAILABLE while the link itself stays ACTIVE.

  • Deleting and recreating a subject: A hard delete on the source replicates on the next sync, not instantly. Wait until the subject is removed from the shadow cluster before registering a new schema under the same name, as recreating it too soon prevents the sync job from detecting the subject/subject-version change.

Configure schema replication

Schema replication is one synchronization task on a shadow link. You configure it by adding the shadow_schema_registry_api option to the schema_registry_sync_options section of the same shadow link that replicates your topic data, consumer offsets, and ACLs. The sections below extend the shadow link configuration described in Configure Shadowing; they do not replace it.

A shadow link whose configuration contains only schema_registry_sync_options replicates schemas and nothing else. For a full migration, keep your topic, consumer offset, and security synchronization options in the same configuration file.

This workflow builds the configuration file that rpk shadow create consumes. The same settings map one-to-one onto the Cloud UI fields, the Control Plane API request, and the Terraform resource shown in Create the shadow link.

The following collapsible sample shows where the schema replication settings sit in a complete shadow link configuration file. The highlighted lines are the schema replication settings, which the sections that follow explain.

Explore a sample configuration file 
# Sample shadow link configuration with API-mode Schema Registry replication

name: confluent-migration                    # Unique name for this shadow link

client_options:
  bootstrap_servers:                         # Source Kafka cluster brokers
  - <source-broker-1>:<port>                 # Example: "pkc-xxxxx.us-east-1.aws.confluent.cloud:9092"
  - <source-broker-2>:<port>
  # For the TLS and authentication settings that the shadow cluster uses to
  # connect to the source Kafka cluster, see the complete configuration file
  # reference in Configure Shadowing.

# Keep the rest of your migration in the same file. Without these sections,
# the link replicates schemas only.
topic_metadata_sync_options:
  # ...your topic replication settings...
consumer_offset_sync_options:
  # ...your consumer offset settings...
security_sync_options:
  # ...your ACL replication settings...

schema_registry_sync_options:
  shadow_schema_registry_api:                # API mode: replicate from a Confluent Schema Registry
    source_url: https://psrc-xxxxx.us-east-1.aws.confluent.cloud   # Source Schema Registry endpoint
    auth_options:
      basic:
        username: <sr-api-key>               # Confluent Schema Registry API key
        password: <sr-api-secret>            # Confluent Schema Registry API secret
    tls_settings:
      enabled: true                          # Use TLS for the connection
    tail_interval: 10s                       # Optional: poll for incremental changes (default: 10s)
    full_sync_interval: 5m                   # Optional: full source scan interval (default: 5m)
    max_source_requests_per_second: 30       # Optional: rate limit for source requests (default: 30)
    source_filter:
      contexts:
      - "."                                  # The default context
      subjects: []                           # Empty: all subjects in the selected contexts
    destination:
      identity: {}                           # Keep source context names
    unsupported_schema_feature_policy: FAIL  # FAIL (default) or REMOVE

For the complete configuration file, including consumer offset and security synchronization options, see Configure Shadowing.

Generate a configuration template

Generate a configuration file template that includes all available fields with comments:

rpk shadow config generate --print-template -o shadow-config-template.yaml

For detailed command options, see rpk shadow config generate.

Connect to the source registry

Configure the connection to the source Schema Registry:

schema_registry_sync_options:
  shadow_schema_registry_api:
    source_url: https://psrc-xxxxx.us-east-1.aws.confluent.cloud  # Source Schema Registry endpoint
    auth_options:
      basic:
        username: <sr-api-key>                # Confluent Schema Registry API key
        password: <sr-api-secret>             # Confluent Schema Registry API secret
    tls_settings:
      enabled: true                           # Use TLS for the connection
    tail_interval: 10s                        # How often to poll for incremental changes
    full_sync_interval: 5m                    # How often to run a full scan
    max_source_requests_per_second: 30        # Rate limit for requests to the source registry

The intervals and rate limit are optional. If you omit them, Redpanda uses the defaults shown above.

To authenticate to the source registry with mTLS instead of HTTP basic authentication, omit auth_options and provide a client certificate and key in tls_settings.

To connect to a source registry that requires no authentication, omit auth_options and do not provide a client certificate.

Select contexts and subjects

By default, the link replicates the entire source registry. To replicate a subset, add a source_filter with the contexts or subjects to include:

schema_registry_sync_options:
  shadow_schema_registry_api:
    # ...connection settings...
    source_filter:
      contexts:
      - ".prod"                               # Replicate this entire context
      subjects:
      - orders-value                          # One subject from the default context

The two lists combine as a union:

  • contexts selects entire contexts: every subject in each listed context replicates.

  • subjects selects individual subjects, using qualified subject syntax: orders-value is the subject in the default context, and :.staging:orders-value is the subject of the same name in the .staging context.

  • When both lists are set, the link replicates everything selected by either list. A subject selected by both lists replicates once.

For example, the preceding filter replicates every subject in the .prod context, plus the single orders-value subject from the default context.

The contexts and subjects lists accept literal names only. Wildcard and prefix patterns are not supported.

Schema Registry contexts provide independent namespaces for subjects within one registry. The default context is named .. For more information, see Schema Registry Contexts.

Map source contexts to destination contexts

Choose how replicated contexts are named on the shadow cluster:

  • identity: Keep the source context names (default behavior for migrations).

  • exact: Map each source context to a different destination context.

schema_registry_sync_options:
  shadow_schema_registry_api:
    # ...connection settings and filters...
    destination:
      identity: {}                            # Keep source context names

To rename contexts during replication:

schema_registry_sync_options:
  shadow_schema_registry_api:
    # ...connection settings and filters...
    destination:
      exact:
        mappings:
        - source: "."                         # Source context
          destination: ".shadow"              # Destination context on the shadow cluster

With exact mapping, the mappings must cover every context that the link replicates. If the link encounters a source context that has no mapping, the schema replication task fails. If new contexts might be created on the source registry after you set up the link, scope the source_filter contexts list to the mapped contexts so that an unexpected context cannot stop replication.

Choose a validation policy

The unsupported_schema_feature_policy setting controls what happens when a source schema uses features that the Redpanda Schema Registry does not support. The unsupported Confluent Schema Registry features are:

  • In schema definitions: rule sets and metadata tags.

  • In subject configurations: override metadata, override rule sets, default metadata, default rule sets, and compatibility groups. Compatibility groups are not the same as compatibility levels, which do replicate.

The policy determines how the sync handles a schema or configuration that uses these features:

Policy Behavior

FAIL (default)

The schema is not replicated. The sync records an error, reports it in the link status, and continues with the remaining schemas.

REMOVE

The unsupported fields are removed and the rest of the schema is imported. The sync records each modification in the link status.

 
schema_registry_sync_options:
  shadow_schema_registry_api:
    # ...connection settings, filters, and destination...
    unsupported_schema_feature_policy: FAIL

Create the shadow link with the method that fits your workflow. All four methods configure the same settings.

  • Cloud UI

  • rpk

  • Control Plane API

  • Terraform

The create shadow link wizard includes a Schema Registry section on the Configuration step:

  1. At the organization level of the Cloud UI, navigate to Shadow Link and click Create shadow link.

  2. Complete the Connection, Shadow, and Source steps as described in Configure Shadowing. For a Confluent source, select Bootstrap URL on the Connection step and enter the Confluent Kafka bootstrap servers.

  3. On the Configuration step, in the Schema Registry section, select the Schema Registry HTTP API mode and configure:

    1. The source Schema Registry URL.

    2. Authentication: none, or HTTP basic. For Confluent Cloud, use a Schema Registry API key and secret.

    3. TLS, with an optional custom CA certificate or an mTLS client certificate.

    4. The scope: every context and subject, or specific ones.

    5. Destination contexts: preserve the source context names, or map each source context to a destination context.

    6. Optionally, the sync behavior: tail interval, full sync interval, maximum source request rate, and the unsupported schema features policy.

  4. The wizard validates the connection details and shows the subjects and versions that match your filters before it creates the link.

  5. Click Create shadow link.

Create the shadow link with your completed configuration file:

rpk shadow create --config-file shadow-config.yaml

For detailed command options, see rpk shadow create.

To change the schema replication settings on an existing link, see rpk shadow update.

To create the link programmatically, include schema_registry_sync_options in the POST /v1/shadow-links request. The following request configures the same API-mode replication as the configuration file above:

curl -X POST 'https://api.redpanda.com/v1/shadow-links' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
  -d '{
    "shadow_link": {
      "shadow_redpanda_id": "<destination-redpanda-cluster-id>",
      "name": "<shadow-link-name>",
      "client_options": {
        "bootstrap_servers": ["<source-broker-1>:<port>"],
        "tls_settings": {
          "enabled": true
        }
      },
      "schema_registry_sync_options": {
        "shadow_schema_registry_api": {
          "source_url": "https://psrc-xxxxx.us-east-1.aws.confluent.cloud",
          "auth_options": {
            "basic": {
              "username": "<sr-api-key>",
              "password": "<sr-api-secret>"
            }
          },
          "tls_settings": {
            "enabled": true
          },
          "source_filter": {
            "contexts": ["."],
            "subjects": []
          },
          "destination": {
            "identity": {}
          },
          "unsupported_schema_feature_policy": "UNSUPPORTED_SCHEMA_FEATURE_POLICY_FAIL"
        }
      }
    }
  }'

For the source cluster connection settings (client_options), authentication, and secret handling, see the Control Plane API tab in Configure Shadowing. For the full request schema, see the Control Plane API reference.

To manage the shadow link declaratively instead of with rpk, use the redpanda_shadow_link resource in the Redpanda Terraform provider. API-mode Schema Registry replication requires provider version 2.3.0 or later.

The configuration keys described on this page map to the resource’s schema_registry_sync_options.shadow_schema_registry_api attribute:

resource "redpanda_shadow_link" "confluent_migration" {
  # Connection settings for the source Kafka cluster (client_options,
  # cluster IDs, and secret handling) are the same as for any shadow link.
  # See the Terraform tab in Configure Shadowing.

  schema_registry_sync_options = {
    shadow_schema_registry_api = {
      source_url = "https://psrc-xxxxx.us-east-1.aws.confluent.cloud"

      auth_options = {
        basic = {
          username = "<sr-api-key>"
          password = "$${secrets.${redpanda_secret.sr_api_secret.name}}"
        }
      }

      tls_settings = {
        enabled = true
      }

      source_filter = {
        contexts = ["."]
      }

      destination = {
        identity = true
      }

      unsupported_schema_feature_policy = "FAIL"
    }
  }
}

Replace the placeholders with your own values:

  • <sr-api-key>: Confluent Schema Registry API key.

  • The password value references a redpanda_secret resource named sr_api_secret that stores the Confluent Schema Registry API secret in the shadow cluster’s secret store. The doubled dollar sign ($$) keeps Terraform from interpolating the ${secrets…​.} reference, which Redpanda resolves at connection time.

Initialize the working directory if you have not already done so, review the plan, and apply the configuration:

terraform init
terraform plan
terraform apply

The attribute shapes differ from the YAML configuration file in one place: destination takes identity = true to preserve source context names, or an exact block with explicit source-to-destination mappings.

For the full attribute schema, see the redpanda_shadow_link reference. For a complete working configuration, including the source cluster connection and secret resources, see the provider example and the Terraform tab in Configure Shadowing.

Verify the configuration

Confirm that the link is configured for API-based schema replication:

rpk shadow describe <link-name> --print-registry

The --print-registry flag prints the Schema Registry section, which includes the shadowing mode, source URL, sync intervals, validation policy, and your context and subject filters. Without it, rpk shadow describe prints only the overview and client sections. For detailed command options, see rpk shadow describe.

Monitor replication status

Check schema replication progress and errors for a link:

rpk shadow status <link-name>

The Schema Registry section of the output reports:

Field Description

Inventory

The number of selected subjects and subject versions on the source, compared with the number of subjects and versions on the destination. The registries are synchronized when the destination counts match the selected source counts. A destination that is behind the source indicates that replication is still in progress.

Current sync

The type of the sync in progress (FULL or TAIL) and the number of subject versions, compatibility configurations, and modes it has changed, including how many unsupported features were removed and how many errors occurred.

Last full sync

Start time, finish time, and change counts for the most recent completed full sync.

Totals since task start

Cumulative change and error counts since the schema replication task started.

Last error

The most recent replication error. With the FAIL validation policy, schemas that fail validation appear here.

The schema replication task runs on the broker and shard that hosts the leader of the _schemas topic’s partition. The counters in the status output are not persisted: expect them to reset to zero when that broker restarts or when leadership of the _schemas partition moves to another broker.

For detailed command options, see rpk shadow status. For general link monitoring, see Monitor Shadowing.

Fail over

For best results, shadow links should be failed over as a single unit, which prevents partial failover scenarios and unexpected results. Selective failover, such as topics only, does not stop schema replication. As part of failover, pause the schema replication task by setting paused: true in the schema_registry_sync_options section of the link configuration. Pausing the task stops further syncing from the source registry and makes the write-blocked destination contexts writable, so your applications can register new schemas on the promoted cluster.

For the complete failover procedure, see Failover.

Next steps