Cloud

Configure Shadowing

You can create and manage shadow links with the Redpanda Cloud UI, the Cloud API, or rpk, giving you flexibility in how you interact with your disaster recovery infrastructure.

Deploy clusters in different geographic regions to protect against regional disasters.

Prerequisites

License and cluster requirements

The shadow cluster must be a BYOC or Dedicated cluster running Redpanda version 25.3 or later. API-mode Schema Registry replication requires version 26.2 or later.
The source cluster can be another Redpanda cluster, or any Kafka API-compatible cluster such as Apache Kafka, Confluent Cloud, or Confluent Platform. See Non-Redpanda source clusters.
Role synchronization requires Redpanda version 26.2 or later on both the source and shadow clusters. See Role filtering.
To configure role synchronization with rpk, use rpk version 26.2.2 or later.

Non-Redpanda source clusters

The shadow cluster reads from the source cluster through the Kafka API, so the source does not need to be a Redpanda cluster. From any Kafka API-compatible source, Shadowing replicates:

Topic data, with preserved offsets and timestamps
Topic configurations, following the rules in Topic properties replication
Consumer group offsets
ACLs

The following require a Redpanda source cluster:

RBAC roles: If the source is not a Redpanda cluster, leave role_sync_options unconfigured. A Roles Migrator task pointed at a non-Redpanda source reports LINK_UNAVAILABLE while the rest of the shadow link keeps running. See Role filtering.
Byte-for-byte Schema Registry replication: Topic mode (shadow_schema_registry_topic) shadows Redpanda’s internal _schemas topic. To replicate schemas from a Confluent Schema Registry or another Confluent-compatible registry, use API mode (shadow_schema_registry_api) instead. See Migrate Schemas from Confluent Schema Registry.

The source cluster must accept one of the authentication methods that the shadow link supports. See Network and authentication.

Amazon MSK can be a source cluster when SASL/SCRAM-SHA-512 authentication is enabled on the MSK cluster. MSK IAM authentication is not supported. AWS Glue Schema Registry is not supported as a schema source.

Cluster configuration

The shadow cluster must have the enable_shadow_linking cluster property set to true.

Starting with Redpanda v25.3, this cluster property is enabled by default on new Redpanda Cloud clusters. For existing clusters on versions earlier than v25.3, you must enable this property manually. See Configure Cluster Properties.

Replication service permissions

You must configure a service account on the source cluster with the following ACL permissions for shadow link replication:

Topics: read permission on all topics you want to replicate
Topic configurations: describe_configs permission on topics for configuration synchronization
Consumer groups: describe and read permission on consumer groups for offset replication
ACLs: describe permission on ACL resources to replicate security policies
Cluster: describe permission on the cluster resource to access ACLs

This service account authenticates from the shadow cluster to the source cluster and performs the actual data replication. The credentials for this account are provided when you set up the shadow link.

Network and authentication

You must configure network connectivity between clusters with appropriate firewall rules to allow the shadow cluster to connect to the source cluster for data replication. Shadowing uses a pull-based architecture where the shadow cluster fetches data from the source cluster. For detailed networking configuration, see Networking.

If using authentication for the shadow link connection, configure the source cluster with your chosen authentication method (SASL/SCRAM, SASL/PLAIN, TLS, mTLS) and ensure the shadow cluster has the proper credentials to authenticate to the source cluster.

Set up Shadowing

To set up Shadowing, you need to create a shadow link and configure filters to select which topics, consumer groups, ACLs, and Schema Registry data to replicate.

If using the Cloud API to set up Shadowing, you must authenticate to the API by including an access token in your requests.

Create a shadow link

Any BYOC or Dedicated cluster can create a shadow link to a source cluster.

You can use rpk to generate a sample configuration file with common filter patterns:

# Generate a sample configuration file with placeholder values
rpk shadow config generate --for-cloud -o shadow-config.yaml

This creates a complete YAML configuration file that you can customize for your environment. The template includes all available fields with comments explaining their purpose. For detailed command options, see rpk shadow config generate --for-cloud.

Explore the configuration file

# Sample ShadowLinkConfig YAML with all fields

name: <shadow-link-name>            # Unique name for this shadow link, example: "production-dr"

cloud_options:
# Use either source_redpanda_id or bootstrap_servers: only one is required.
  source_redpanda_id: <source-cluster-id>    # Optional: 20 character lowercase ID of the cluster
                                             # Example: m7xtv2qq5njbhwruk88f
  shadow_redpanda_id: <shadow-cluster-id>    # 20 character lowercase ID of the cluster
                                             # Example: m7xtv2qq5njbhwruk88f
client_options:
  bootstrap_servers:                         # Source cluster brokers to connect to
  - <source-broker-1>:<port>                 # Example: "prod-kafka-1.example.com:9092"
  - <source-broker-2>:<port>                 # Example: "prod-kafka-2.example.com:9092"
  - <source-broker-3>:<port>                 # Example: "prod-kafka-3.example.com:9092"
  source_cluster_id: <cluster-id>            # Optional: UUID assigned by Redpanda
                                             # Example: a882bc98-7aca-40f6-a657-36a0b4daf1fd
# This UUID is not available in Redpanda Cloud.

  # TLS settings using PEM strings
  tls_settings:
    enabled: true
    tls_pem_settings:
      ca: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      key: ${secrets.<key-from-shadow-cluster-secret-store>}
      cert: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----

# Create SASL credentials in the source cluster.
# Then, with this configuration, ensure the shadow cluster uses the credentials
# to authenticate to the source cluster.
  authentication_configuration:
    # SASL/SCRAM authentication
    scram_configuration:
      username: <sasl-username>       # SASL/SCRAM username, example: "shadow-replication-user"
      password: ${secrets.<sasl-password-secret-id>}       # ID of secret containing SASL/SCRAM password
      scram_mechanism: SCRAM_SHA_256  # SCRAM mechanism: "SCRAM_SHA_256" or "SCRAM_SHA_512"
    # SASL/PLAIN authentication: to use SASL/PLAIN instead of SASL/SCRAM,
    # remove scram_configuration and uncomment the following lines.
    # plain_configuration:
    #   username: <sasl-username>     # SASL/PLAIN username, example: "shadow-replication-user"
    #   password: ${secrets.<sasl-password-secret-id>}  # ID of secret containing SASL/PLAIN password

  # Connection tuning - adjust based on network characteristics
  metadata_max_age_ms: 10000          # How often to refresh cluster metadata (default: 10000ms)
  connection_timeout_ms: 1000         # Connection timeout (default: 1000ms, increase for high latency)
  retry_backoff_ms: 100               # Backoff between retries (default: 100ms)
  fetch_wait_max_ms: 500              # Max time to wait for fetch requests (default: 500ms)
  fetch_min_bytes: 5242880            # Min bytes per fetch (default: 5MB)
  fetch_max_bytes: 20971520           # Max bytes per fetch (default: 20MB)
  fetch_partition_max_bytes: 5242880  # Max bytes per partition fetch (default: 5MB)

topic_metadata_sync_options:
  interval: 30s                       # How often to sync topic metadata (examples: "30s", "1m", "5m")
  auto_create_shadow_topic_filters:   # Filters for automatic topic creation
  - pattern_type: LITERAL             # Include all topics (wildcard)
    filter_type: INCLUDE
    name: '*'
  - pattern_type: PREFIX              # Exclude topics with specific prefix
    filter_type: EXCLUDE
    name: <topic-prefix-to-exclude>   # Examples: "temp-", "test-", "debug-"
  synced_shadow_topic_properties:     # Additional topic properties to sync (beyond defaults)
  - retention.ms                      # Topic retention time
  - segment.ms                        # Segment roll time
  exclude_default: false              # Include default properties (compression, retention, etc.)
  start_at_earliest: {}               # Start from the beginning of source topics (default)
  paused: false                       # Enable topic metadata synchronization

consumer_offset_sync_options:
  interval: 30s                       # How often to sync consumer group offsets
  paused: false                       # Enable consumer offset synchronization
  group_filters:                      # Filters for consumer groups to sync
  - pattern_type: LITERAL
    filter_type: INCLUDE
    name: '*'                         # Include all consumer groups

security_sync_options:
  interval: 30s                       # How often to sync security settings
  paused: false                       # Enable security settings synchronization
  acl_filters:                        # Filters for ACLs to sync
  - resource_filter:
      resource_type: TOPIC            # Resource type: "TOPIC", "GROUP", "CLUSTER"
      pattern_type: PREFIXED          # Pattern type: "LITERAL", "PREFIXED"
      name: <resource-pattern>        # Examples: "prod-", "app-data-"
    access_filter:
      principal: User:<username>      # Principal name, example: "User:app-service"
      operation: ANY                  # Operation: "READ", "WRITE", "CREATE", "DELETE", "ALTER", "DESCRIBE", "ANY"
      permission_type: ALLOW          # Permission: "ALLOW" or "DENY"
      host: '*'                       # Host pattern, examples: "*", "10.0.0.0/8", "app-server.example.com"

schema_registry_sync_options:         # Schema Registry synchronization options
  shadow_schema_registry_topic: {}    # Byte-for-byte _schemas replication (Redpanda source)
  # For a Confluent source, replace the line above with a
  # shadow_schema_registry_api block (API mode). See Migrate Schemas
  # from Confluent Schema Registry.

# Role synchronization requires a Redpanda source cluster and v26.2 or
# later on both clusters. Omit role_sync_options otherwise.
role_sync_options:                    # RBAC role synchronization options
  interval: 30s                       # How often to sync roles
  paused: false                       # Enable role synchronization
  role_name_filters:                  # Default empty: no roles sync until you add an INCLUDE filter
  - pattern_type: LITERAL             # Pattern type: "LITERAL", "PREFIX"
    filter_type: INCLUDE              # Filter type: "INCLUDE", "EXCLUDE"
    name: '*'                         # Include all roles ('*' must be the only character, LITERAL only)

Because the shadow cluster pulls from the source cluster, the shadow cluster requires credentials to connect to the source cluster. And because you cannot store plaintext passwords in Redpanda Cloud, you must create a secret to hold the password for the user on the source cluster. If using mTLS, you must also create a secret to hold the key of the client certificate for the client to authenticate. Reference that secret in client_options.tls_settings.tls_pem_settings.key in the configuration file.

In the shadow cluster, create the secret:
- Cloud UI
- rpk
- Data Plane API
- Terraform
If the source is a Redpanda Cloud cluster and the source cluster user does not exist yet, create it first: in the source cluster, go to Security > Users, click Create user, and copy the generated password before you close the dialog. Then click Add permissions and grant the permissions in Replication service permissions (or, for a test link, allow all operations). For other source clusters, create the user and grant the Replication service permissions with your platform’s tools.

In the shadow cluster, go to the Secrets Store page and click Create Secret. Enter an ID that starts with an uppercase letter and contains only uppercase letters, numbers, and underscores, for example SOURCE_SASL_PASSWORD. The Secrets Store form also accepts slashes and hyphens, but the shadow link wizard rejects IDs that contain them. Paste the source cluster user’s password as the value, and make sure Redpanda Cluster is one of the selected scopes. You enter this secret ID, not the password, when you create the shadow link.

In the shadow cluster, create a secret to store the authentication credential that the cluster will use (the password of scram_configuration or plain_configuration in the example configuration in the next step). Your secret must be scoped to "Redpanda Cluster".

Use rpk security secret create to create the secret from the command line.

In the shadow cluster, create a secret to store the authentication credential that the cluster will use (the password of scram_configuration or plain_configuration in the example configuration in the next step). Your secret must be scoped to "Redpanda Cluster".

Use the Data Plane API to programmatically create the secret.

With Terraform, you define the secret and the shadow link together: the redpanda_secret resource in the next step’s Terraform tab creates this secret as part of the same configuration, so there is nothing to do in this step.

In the shadow cluster, create a shadow link to the source cluster.

Cloud UI
rpk
Control Plane API
Terraform

At the organization level of the Cloud UI, navigate to Shadow Link. You can also start from a cluster’s Shadow Links page by clicking Create shadow link in Redpanda Cloud.
Click Create shadow link. The Cloud UI guides you through the following steps.
On the Connection step:
1. Enter a unique name for the shadow link. The name must start and end with lowercase alphanumeric characters, hyphens allowed.
2. Select where the replicated data comes from: Redpanda Cloud cluster for an existing Redpanda Cloud cluster, or Bootstrap URL to connect to any Kafka-compatible cluster.
3. Configure TLS for the connection. Enable TLS is on by default. Optionally, expand Use a custom CA certificate to add the CA for a self-signed or private certificate on the source cluster, or Use mutual TLS (mTLS) to add a client certificate.
4. Under Authentication method, select SCRAM or PLAIN. Enter the Username of the source cluster user and, in the Password field, the ID of the secret that contains the password, created in the previous step. For SCRAM, also select the SASL mechanism. SASL/PLAIN sends the username and password unhashed, so use it only over a TLS-encrypted connection. SASL/SCRAM is the recommended method.
5. Optionally, expand Advanced options to configure client connection settings.
On the Shadow step, select the shadow cluster that receives the replicated data.
On the Source step, select the source cluster. This step appears only when the source is an existing Redpanda Cloud cluster. If you selected Bootstrap URL, the wizard skips this step.
On the Configuration step, specify what to shadow from the source cluster: topics, ACLs, roles, consumer groups, and Schema Registry data. The Roles section appears only when the shadow cluster runs Redpanda version 26.2 or later. See Role filtering.

For a Confluent source, configure the Shadow Schema Registry section as described in Migrate Schemas from Confluent Schema Registry.
Click Create shadow link.

Run rpk cloud login. Select your shadow cluster when prompted.
To create a shadow link with the source cluster using rpk, run the following command from the shadow cluster:

# When logged in, optionally create a new rpk profile to easily
# switch to the shadow cluster
rpk profile create --from-cloud <shadow-cluster-id> shadow-cluster

# Use the generated configuration file to create the shadow link
rpk shadow create --config-file shadow-config.yaml

For detailed command options, see rpk shadow create.

Use rpk profile to save your cluster connection details and credentials for both source and shadow clusters. This allows you to easily switch between the two configurations.

To create a shadow link using the Control Plane API, make a POST /shadow-links request from the shadow cluster:

curl -X POST 'https://api.redpanda.com/v1/shadow-links' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
  -d '{
    "shadow_link": {
      "shadow_redpanda_id": "<destination-redpanda-cluster-id>",
      "name": "<shadow-link-name>",
      "client_options": {
        "bootstrap_servers": ["<source-broker-1>:<port>", "<source-broker-2>:<port>", "<source-broker-3>:<port>"],
        "tls_settings": {
          "enabled": true
        },
        "authentication_configuration": {
          "scram_configuration": {
            "username": "<sasl-username>",
            "password": "${secrets.<sasl-password-secret-id>}",
            "scram_mechanism": "SCRAM_MECHANISM_SCRAM_SHA_256"
          }
        }
      },
      "topic_metadata_sync_options": {
        "interval": "30s",
        "auto_create_shadow_topic_filters": [
          {
            "name": "*",
            "filter_type": "FILTER_TYPE_INCLUDE",
            "pattern_type": "PATTERN_TYPE_LITERAL"
          },
          {
            "name": "<topic-prefix-to-exclude>",
            "filter_type": "FILTER_TYPE_EXCLUDE",
            "pattern_type": "PATTERN_TYPE_PREFIX"
          }
        ],
        "start_at_earliest": {},
        "paused": false
      },
      "consumer_offset_sync_options": {
        "paused": true
      },
      "security_sync_options": {
        "paused": true
      }
    }
  }'

Replace the placeholders with your own values:

<destination-redpanda-cluster-id>: ID of the shadow (destination) cluster.
<shadow-link-name>: Unique name for this shadow link, for example, production-dr.
<source-broker-1>:<port>, <source-broker-2>:<port> …: Source cluster brokers to connect to, for example, prod-kafka-1.example.com:9092, prod-kafka-2.example.com:9092.
<sasl-username>: SASL username, for example, shadow-replication-user. You create this user in the source cluster.
<sasl-password-secret-id>: The ID of the secret containing the SASL password from the source cluster.
<topic-prefix-to-exclude>: Exclude topics that use this prefix, for example, temp-, test-, debug-.

The response object represents the long-running operation of creating a shadow link.

To authenticate with SASL/PLAIN instead of SASL/SCRAM, replace the scram_configuration object with a plain_configuration object that contains username and password. As with SCRAM, the password must be a secret reference in the form ${secrets.<sasl-password-secret-id>}; the API rejects a plaintext password.

To replicate RBAC roles, add a role_sync_options object. See Role filtering.

To include Schema Registry replication in the request, add a schema_registry_sync_options object; for the API-mode request used to replicate from a Confluent Schema Registry, see Migrate Schemas from Confluent Schema Registry. For the full API reference, see Control Plane API reference.

Manage shadow links declaratively with the redpanda_shadow_link resource in the Redpanda Terraform provider (version 2.0.0 or later). The resource supports the full shadow link lifecycle: create, update, import, and destroy.

Store the SASL/SCRAM password for the source cluster user in the shadow cluster’s secret store with the redpanda_secret resource (see Manage cluster secrets for naming, write-only behavior, and rotation), then reference the secret from the shadow link configuration:

resource "redpanda_secret" "source_password" {
  name                = "<secret-name>"
  secret_data         = var.source_user_password
  secret_data_version = 1   # Increment when you rotate the password.
  scopes              = ["SCOPE_REDPANDA_CLUSTER"]
  cluster_api_url     = redpanda_cluster.shadow.cluster_api_url
}

resource "redpanda_shadow_link" "production_dr" {
  name               = "<shadow-link-name>"
  shadow_redpanda_id = "<destination-redpanda-cluster-id>"
  source_redpanda_id = "<source-redpanda-cluster-id>"

  client_options = {
    tls_settings = {
      enabled = true
    }
    authentication_configuration = {
      scram_configuration = {
        scram_mechanism = "SCRAM_SHA_256"
        username        = "<sasl-username>"
        password        = "$${secrets.${redpanda_secret.source_password.name}}"
      }
    }
  }
}

Replace the placeholders with your own values:

<secret-name>: Name for the secret that stores the SASL/SCRAM password. Must be uppercase, matching ^[A-Z][A-Z0-9_]*$, for example, SOURCE_SCRAM_PASSWORD.
<shadow-link-name>: Unique name for this shadow link, for example, production-dr.
<destination-redpanda-cluster-id>: ID of the shadow (destination) cluster.
<source-redpanda-cluster-id>: ID of the source cluster. For a source cluster that is not managed in Redpanda Cloud, omit source_redpanda_id and set client_options.bootstrap_servers instead.
<sasl-username>: SASL username, for example, shadow-replication-user. You create this user in the source cluster, and you can manage it with the redpanda_user resource. Grant it the permissions listed in Replication service permissions, which you can manage with the redpanda_acl resource.

To configure topic, consumer group, ACL, Schema Registry, and role synchronization, add the corresponding option blocks (topic_metadata_sync_options, consumer_offset_sync_options, security_sync_options, schema_registry_sync_options, role_sync_options) to the resource. The role_sync_options block requires provider version 2.3.0 or later. To authenticate with SASL/PLAIN, use a plain_configuration block in place of scram_configuration. For the shadow_schema_registry_api block used to replicate from a Confluent Schema Registry, see Migrate Schemas from Confluent Schema Registry. For the full schema and a complete working example, see the redpanda_shadow_link reference and the provider example.

If you manage the shadow cluster with the redpanda_cluster resource, you can set the enable_shadow_linking cluster property in cluster_configuration.custom_properties_json.

To protect against accidental deletion, Terraform refuses to destroy a shadow link unless the resource sets allow_deletion = true.

Set filters

Filters determine which resources Shadowing automatically creates when establishing your shadow link.

Topic filters select which topics Shadowing automatically creates as shadow topics when they appear on the source cluster. After Shadowing creates a shadow topic, it continues replicating until you failover the topic, delete it, or delete the entire shadow link.

Consumer group and ACL filters control which groups and security policies replicate to maintain application functionality.

Filter types and patterns

Each filter uses two key settings:

Pattern type: Determines how names are matched
- LITERAL: Matches names exactly (including the special wildcard * to match all items)
- PREFIX: Matches names that start with the specified string
Filter type: Specifies whether to INCLUDE or EXCLUDE matching items
- INCLUDE: Replicate items that match the pattern
- EXCLUDE: Skip items that match the pattern

Filter processing rules

Redpanda processes filters in the order you define them with EXCLUDE filters taking precedence. Design your filter lists carefully:

Exclude filters win: If any EXCLUDE filter matches a resource, it is excluded regardless of INCLUDE filters.
Order matters for INCLUDE filters: Among INCLUDE filters, the first match determines the result.
Default behavior: Items that don’t match any filter are excluded from replication.

Common filtering patterns

Replicate all topics except test topics:

topic_metadata_sync_options:
  auto_create_shadow_topic_filters:
  - pattern_type: PREFIX
    filter_type: EXCLUDE
    name: test-                        # Exclude all test topics
  - pattern_type: LITERAL
    filter_type: INCLUDE
    name: '*'                         # Include all other topics

Replicate only production topics:

topic_metadata_sync_options:
  auto_create_shadow_topic_filters:
  - pattern_type: PREFIX
    filter_type: INCLUDE
    name: prod-                       # Include production topics
  - pattern_type: PREFIX
    filter_type: INCLUDE
    name: production-                 # Alternative production prefix

Replicate specific consumer groups:

consumer_offset_sync_options:
  group_filters:
  - pattern_type: LITERAL
    filter_type: INCLUDE
    name: critical-app-consumers      # Include specific consumer group
  - pattern_type: PREFIX
    filter_type: INCLUDE
    name: prod-consumer-              # Include production consumers

Schema Registry synchronization

Shadowing can replicate Schema Registry data in one of two modes:

Topic mode (shadow_schema_registry_topic): Shadows the _schemas system topic for byte-for-byte replication of schema definitions, versions, and compatibility settings from another Redpanda cluster.
API mode (shadow_schema_registry_api): Polls the source Schema Registry over HTTP and imports selected contexts and subjects, with validation. Use this mode to replicate schemas from a Confluent Schema Registry, or to replicate only part of the source registry. See Migrate Schemas from Confluent Schema Registry.

A shadow link uses one mode or the other, not both.

To enable topic mode, add the following to your shadow link configuration:

schema_registry_sync_options:
  shadow_schema_registry_topic: {}

To enable API mode instead, add a shadow_schema_registry_api block; its connection, filtering, and mapping options are described in Migrate Schemas from Confluent Schema Registry.

Topic mode requirements:

The _schemas topic must exist on the source cluster
The _schemas topic must not exist on the shadow cluster, or must be empty
Once enabled, the _schemas topic will be replicated completely

Important: After the _schemas topic becomes a shadow topic, it cannot be stopped without either failing over the topic or deleting it entirely.

System topic filtering rules

Redpanda system topics have the following specific filtering restrictions:

Literal filters for __consumer_offsets and _redpanda.audit_log are rejected.
Prefix filters for topics starting with _redpanda or __redpanda are rejected.
Wildcard * filters will not match topics that start with _redpanda or __redpanda.
To shadow specific system topics, you must provide explicit literal filters for those individual topics.

ACL filtering

ACLs are replicated by the Security Migrator task. This is recommended to ensure that your shadow cluster has the same permissions as your source cluster. To configure ACL filters:

security_sync_options:
  acl_filters:
  # Include read permissions for production topics
  - resource_filter:
      resource_type: TOPIC            # Filter by topic resource
      pattern_type: PREFIXED          # Match by prefix
      name: prod-                     # Production topic prefix
    access_filter:
      principal: User:app-user        # Application service user
      operation: READ                 # Read operation
      permission_type: ALLOW          # Allow permission
      host: '*'                       # Any host
  # Include consumer group permissions
  - resource_filter:
      resource_type: GROUP            # Filter by consumer group
      pattern_type: LITERAL           # Exact match
      name: '*'                       # All consumer groups
    access_filter:
      principal: User:app-user        # Same application user
      operation: READ                 # Read operation
      permission_type: ALLOW          # Allow permission
      host: '*'                       # Any host

Role filtering

After a failover, your users and applications need the same permissions on the shadow cluster that they had on the source. Role synchronization prepares the shadow cluster for that moment: the Roles Migrator task periodically reads RBAC role definitions and role memberships from the source cluster and mirrors them to the shadow cluster. Use role synchronization together with ACL filters whenever your authorization model uses roles, so that role-based permissions keep working after failover.

In the configuration file, rpk, the Control Plane API, and Terraform, no roles are replicated by default: the role_name_filters list defaults to empty, which synchronizes nothing. Add at least one INCLUDE filter to start replicating roles. When you create a shadow link in the Cloud UI, the default is All roles.

To replicate all roles:

role_sync_options:
  interval: 30s                       # How often to sync roles
  role_name_filters:
  - pattern_type: LITERAL
    filter_type: INCLUDE
    name: '*'                         # '*' must be the only character and requires LITERAL

Role filters use the same pattern and filter types as topic filters: pattern_type is LITERAL or PREFIX, and filter_type is INCLUDE or EXCLUDE.

When you create a shadow link in the Cloud UI, configure role filters in the Roles section of the Configuration step:

All roles: Replicates every role. This is equivalent to a single INCLUDE filter with the literal name *, and it is the default.
Specify roles: Replicates only the roles that match the filters you add. Each filter takes a role name and a match mode: Include specific roles, Include starting with, Exclude specific, or Exclude starting with. The four modes correspond to the filter_type and pattern_type combinations above. Click Add filter to add another filter.

The Roles section appears only when the shadow cluster runs Redpanda version 26.2 or later. If the shadow cluster’s version does not support role synchronization, the Control Plane API rejects a request that sets any role_sync_options field.

The Cloud UI does not expose the interval or paused settings: a link created in the Cloud UI uses the default sync interval, and pausing role synchronization requires role_sync_options in the configuration file with rpk, the Control Plane API, or Terraform.

To change role filters on an existing link, select the link, click Edit, and use the Roles card on the Shadowing tab. When the link has no role synchronization configured, the card starts on Specify roles with no filters, and role synchronization stays off unless you add a filter or select All roles.

Within the configured filters, the Roles Migrator task keeps the shadow cluster’s roles as a full mirror of the source:

The task creates roles that exist on the source but not on the shadow.
The task updates roles whose membership changed on the source.
The task deletes roles that were deleted on the source.

The task does not modify roles on the shadow cluster that fall outside your filters.

Role synchronization replicates role definitions and memberships only:

Shadowing does not replicate user credentials. Manage and distribute credentials outside of Shadowing, and provision users on the shadow cluster before failover.
Role synchronization does not replicate the ACLs bound to roles. Configure ACL filters as well, so that both the roles and the permissions granted to them exist on the shadow cluster.

Role synchronization uses the same connection and service account as the rest of the shadow link. The cluster describe permission already required for ACL replication also covers role reads, and no additional network configuration is required. Both the source and shadow clusters must be running Redpanda 26.2 or later. If the source is not a Redpanda cluster, or is a Redpanda cluster running a version earlier than 26.2, the Roles Migrator task enters the LINK_UNAVAILABLE state, without affecting the other tasks on the link.

Role synchronization continues to run after a failover. Failover acts on topic data only: it stops topic replication and makes shadow topics writable, but the sync tasks, including the Roles Migrator, keep running on their intervals. To stop role synchronization, set paused: true in role_sync_options, or delete the shadow link.

To monitor role synchronization, run rpk shadow status. The Roles Migrator task’s Reason field reports the counts from the most recent sync cycle, for example: Synced roles: 2 created, 0 updated, 0 deleted, 0 failures. In the Cloud UI, the Tasks tab of the shadow link also shows the status of the Roles Migrator task.

Consumer group filtering and behavior

Consumer group filters determine which consumer groups have their offsets replicated to the shadow cluster by the Consumer Group Shadowing task.

Offset replication operates selectively within each consumer group. Only committed offsets for active shadow topics are synchronized, even if the consumer group has offsets for additional topics that aren’t being shadowed. For example, if consumer group "app-consumers" has committed offsets for "orders", "payments", and "inventory" topics, but only "orders" is an active shadow topic, then only the "orders" offsets will be replicated to the shadow cluster.

consumer_offset_sync_options:
  interval: 30s                      # How often to sync consumer group offsets
  paused: false                       # Enable consumer offset synchronization
  group_filters:
  - pattern_type: PREFIX
    filter_type: INCLUDE
    name: prod-consumer-              # Include production consumer groups
  - pattern_type: LITERAL
    filter_type: EXCLUDE
    name: test-consumer-group         # Exclude specific test groups

Important consumer group considerations

Avoid name conflicts: If you plan to consume data from the shadow cluster, do not use the same consumer group names as those used on the source cluster. While this won’t break shadow linking, it can impact your RPO/RTO because conflicting group names may interfere with offset replication and consumer resumption during disaster recovery.

Offset clamping: When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the high watermark (HWM) of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition’s HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster.

Starting offset for new shadow topics

When the Source Topic Sync task creates a shadow topic for the first time, you can control where replication begins on the source topic. This setting only applies to empty shadow partitions and is crucial for disaster recovery planning. Changing this configuration only affects new shadow topics, existing shadow topics continue replicating from their current position.

topic_metadata_sync_options:
  start_at_earliest: {}

Alternatively, to start from the most recent offset:

topic_metadata_sync_options:
  start_at_latest: {}

Or to start from a specific timestamp:

topic_metadata_sync_options:
  start_at_timestamp: 2024-01-01T00:00:00Z

Starting offset options:

earliest (default): This replicates all existing data from the source topic. Use this for complete disaster recovery where you need full data history.
latest: This starts replication from the current end of the source topic, skipping existing data. Use this when you only need new data for disaster recovery and want to minimize initial replication time.
timestamp: This starts replication from the first record with a timestamp at or after the specified time. Use this for point-in-time disaster recovery scenarios.

The starting offset only affects new shadow topics. After a shadow topic exists and has data, changing this setting has no effect on that topic’s replication.

Networking

Configure network connectivity between your source and shadow clusters to enable shadow link replication. The shadow cluster initiates connections to the source cluster using a pull-based architecture.

For additional details about networking, see Network and authentication.

Connection requirements

Direction: Shadow cluster connects to source cluster (outbound from shadow, inbound to source)
Protocol: Kafka protocol over TCP (default port 9092, or your configured listener ports)
Persistence: Connections remain active for continuous replication

Firewall configuration

You must configure firewall rules to allow the shadow cluster to reach the source cluster.

On the source cluster network:

Allow inbound TCP connections on Kafka listener ports (typically 9092).
Allow connections from the shadow cluster’s IP addresses or subnets.

On the shadow cluster network:

Allow outbound TCP connections to the source cluster’s Kafka listener ports.
Ensure DNS resolution works for source cluster hostnames.

Bootstrap servers

Specify multiple bootstrap servers in your shadow link configuration for high availability:

client_options:
  bootstrap_servers:                  # Source cluster brokers to connect to
  - <source-broker-1>:<port>          # Example: "prod-kafka-1.example.com:9092"
  - <source-broker-2>:<port>          # Example: "prod-kafka-2.example.com:9092"
  - <source-broker-3>:<port>          # Example: "prod-kafka-3.example.com:9092"

The shadow cluster uses these addresses to discover all brokers in the source cluster. If one bootstrap server is unavailable, the shadow cluster tries the next one in the list.

Network security

For production deployments, secure the network connection between clusters:

TLS encryption:

client_options:
  tls_settings:
    enabled: true                     # Enable TLS
    tls_pem_settings:
      ca: |-                          # CA certificate in PEM format
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      key: ${secrets.<key-from-shadow-cluster-secret-store>}  # Client private key (can use secrets reference)
      cert: |-                        # Optional: Client certificate in PEM format for mutual TLS
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
    do_not_set_sni_hostname: false   # Optional: Skip SNI hostname when using TLS (default: false)

Authentication:

client_options:
  authentication_configuration:
    # SASL/SCRAM authentication.
    # Create SASL credentials in the source cluster.
    # Then, with this configuration, ensure the shadow cluster uses the credentials
    # to authenticate to the source cluster.
    scram_configuration:
      username: <sasl-username>       # SASL/SCRAM username, example: "shadow-replication-user"
      password: ${secrets.<sasl-password-secret-id>}       # ID of secret containing SASL/SCRAM password
      scram_mechanism: SCRAM_SHA_256  # SCRAM mechanism: "SCRAM_SHA_256" or "SCRAM_SHA_512"
    # SASL/PLAIN authentication: to use SASL/PLAIN instead of SASL/SCRAM,
    # remove scram_configuration and uncomment the following lines.
    # plain_configuration:
    #   username: <sasl-username>     # SASL/PLAIN username, example: "shadow-replication-user"
    #   password: ${secrets.<sasl-password-secret-id>}  # ID of secret containing SASL/PLAIN password

Connection tuning

Adjust connection parameters based on your network characteristics. For example:

client_options:
  # Connection and metadata settings
  connection_timeout_ms: 1000         # Default 1000ms, increase for high-latency networks
  retry_backoff_ms: 100               # Default 100ms, backoff between connection retries
  metadata_max_age_ms: 10000          # Default 10000ms, how often to refresh cluster metadata

  # Fetch request settings
  fetch_wait_max_ms: 500              # Default 500ms, max time to wait for fetch requests
  fetch_min_bytes: 5242880            # Default 5MB, minimum bytes to fetch per request
  fetch_max_bytes: 20971520           # Default 20MB, maximum bytes to fetch per request
  fetch_partition_max_bytes: 5242880  # Default 5MB, maximum bytes to fetch per partition

Update an existing shadow link

To modify a shadow link configuration after creation, run:

Cloud UI
rpk
Control Plane API
Terraform

At the organization level of the Cloud UI, navigate to Shadow Link.
Select the shadow link you want to modify, and click Edit.
Edit the shadow link settings or the shadowing behavior by specifying which content from the source cluster to shadow (topics, ACLs, roles, consumer groups, Schema Registry). You can also enable additional topic properties to be shadowed or disable optional topic properties from being included in the shadowing.
Click Save to apply changes.

rpk shadow update <shadow-link-name>

For detailed command options, see rpk shadow update.

This opens your default editor to modify the shadow link configuration. The submitted configuration replaces the entire shadow link configuration: any field omitted is reset to its default value. The shadow link name cannot be changed - you must delete and recreate the link to rename it.

curl -X PATCH 'https://api.redpanda.com/v1/shadow-links/<shadow-link-id>' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
  -d '{
      "security_sync_options": {
        "paused": false
      }
  }'

This endpoint returns a long-running operation.

For the full API reference, see Control Plane API reference.

Edit the redpanda_shadow_link resource configuration and run terraform apply. The provider sends only the changed fields to the API. The shadow link name and cluster IDs are immutable: if you change them, Terraform plans a replacement (destroy and recreate) instead of an update. The destroy step of a replacement succeeds only when the resource sets allow_deletion = true, the same guard described in the create step.

Was this helpful?

group Ask in the community

mail Share your feedback

What do you think of this page?

Let us know more:

Let us contact you about your feedback:

Configure Shadowing

Prerequisites

License and cluster requirements

Non-Redpanda source clusters

Cluster configuration

Replication service permissions

Network and authentication

Set up Shadowing

Create a shadow link

Set filters

Filter types and patterns

Filter processing rules

Common filtering patterns

Schema Registry synchronization

System topic filtering rules

ACL filtering

Role filtering

Consumer group filtering and behavior

Important consumer group considerations

Starting offset for new shadow topics

Networking

Connection requirements

Firewall configuration

Bootstrap servers

Network security

Connection tuning

Update an existing shadow link