Shadowing Overview

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

Shadowing is Redpanda’s enterprise-grade disaster recovery solution that establishes asynchronous, offset-preserving replication between two distinct Redpanda clusters. A cluster is able to create a dedicated client that continuously replicates source cluster data, including offsets, timestamps, and cluster metadata. This creates a read-only shadow cluster that you can quickly failover to handle production traffic during a disaster.

See Failover Runbook for immediate step-by-step disaster procedures.

Unlike traditional replication tools that re-produce messages, Shadowing copies data at the byte level, ensuring shadow topics contain identical copies of source topics with preserved offsets and timestamps.

Shadowing replicates:

  • Topic data: All records with preserved offsets and timestamps

  • Topic configurations: Partition counts, retention policies, and other topic properties

  • Consumer group offsets: Enables seamless consumer resumption after failover

  • Access control lists (ACLs): User permissions and security policies

  • Schema Registry data: Schema definitions and compatibility settings

How Shadowing fits into disaster recovery

Shadowing addresses enterprise disaster recovery requirements driven by regulatory compliance and business continuity needs. Organizations typically want to minimize both recovery time objective (RTO) and recovery point objective (RPO), and Shadowing asynchronous replication helps you achieve both goals by reducing data loss during regional outages and enabling rapid application recovery.

The architecture follows an active-passive pattern. The source cluster processes all production traffic while the shadow cluster remains in read-only mode, continuously receiving updates. If a disaster occurs, you can failover the shadow topics using the Admin API or rpk, making them fully writable. At that point, you can redirect your applications to the shadow cluster, which becomes the new production cluster.

Shadowing complements Redpanda’s existing availability and recovery capabilities. High availability actively protects your day-to-day operations, handling reads and writes seamlessly during node or availability zone failures within a region. Shadowing is your safety net for catastrophic regional disasters. While Whole Cluster Restore provides point-in-time recovery from Tiered Storage, Shadowing delivers near real-time, cross-region replication for mission-critical applications that require rapid failover with minimal data loss.

Limitations

Shadowing is designed for active-passive disaster recovery scenarios. Each shadow cluster can maintain only one shadow link.

Shadowing operates exclusively in asynchronous mode and doesn’t support active-active configurations. This means there will always be some replication lag. You cannot write to both clusters simultaneously.

Data transforms are not supported on shadow clusters while Shadowing is active. Writing to shadow topics is blocked.

During a disaster, audit log history from the source cluster is lost, though the shadow cluster begins generating new audit logs immediately after the failover.

After you failover shadow topics, automatic fallback to the original source cluster is not supported.

Best Practices

To ensure reliable disaster recovery with Shadowing:

  • Avoid write caching on source topics: Do not shadow source topics that have write caching enabled. Write caching can result in data loss on the source cluster during broker resets, causing cluster divergence if shadow links replicate data before it’s lost on the source.

  • Do not modify shadow topic properties: Avoid modifying synced topic properties on shadow topics, as these properties automatically revert to source topic values.

Implementation overview

Choose your implementation approach:

Disaster readiness checklist

Before a disaster occurs, ensure you have:

  • Access to shadow cluster administrative credentials

  • Shadow link names and configuration details, and networking documented

  • Application connection strings for the shadow cluster prepared

  • Tested failover procedures in a non-production environment

Next steps

After setting up Shadowing for your Redpanda clusters, consider these additional steps:

  • Test your disaster recovery procedures: Regularly practice failover scenarios in a non-production environment. See Failover Runbook for step-by-step disaster procedures.

  • Monitor shadow link health: Set up alerting on the metrics described above to ensure early detection of replication issues.

  • Implement automated failover: Consider developing automation scripts that can detect outages and initiate failover based on predefined criteria.

  • Review security policies: Ensure your ACL filters replicate the appropriate security settings for your disaster recovery environment.

  • Document your configuration: Maintain up-to-date documentation of your shadow link configuration, including network settings, authentication details, and filter definitions.