Labs

Explore Shadow Linking for Disaster Recovery

Shadow linking is a built-in disaster recovery mechanism that continuously replicates topics, consumer group offsets, and Schema Registry data from a source cluster to a shadow cluster running on Kubernetes. This lab demonstrates how shadow linking works and how you can use it to fail over to a shadow cluster.

In this lab, you deploy two single-node Redpanda clusters on Kubernetes and configure a shadow link between them. You then explore basic topic shadowing, Schema Registry shadowing, and consumer group failover.

Prerequisites

You must have the following:

What gets deployed

The setup script installs the following components:

  • cert-manager (namespace: cert-manager): Manages TLS certificates

  • Redpanda Operator (namespace: rp-operator): Manages Redpanda cluster lifecycle

  • Source cluster (namespace: source): Primary cluster where you write data

  • Shadow cluster (namespace: shadow): Replication target cluster

  • ShadowLink resource: Defines the replication relationship between clusters

The shadow link configuration mirrors all topics using wildcard filtering, replicates consumer group offsets, and synchronizes the Schema Registry, all at 5-second intervals.

Run the lab

  1. Clone this repository:

    git clone https://github.com/redpanda-data/redpanda-labs.git
cd redpanda-labs/kubernetes/shadow-linking

  2. Create a local Kubernetes cluster using kind:

    kind create cluster --name redpanda-shadow

  3. Run the setup script to deploy both clusters and configure shadow linking:

    ./setup.sh

    The script installs cert-manager, deploys the Redpanda Operator, creates the source and shadow clusters, configures the shadow link, sets up port forwarding for local access, and creates rpk profiles for both clusters.

  4. Verify both clusters are healthy:

    rpk --profile source cluster health
rpk --profile shadow cluster health

Explore basic topic shadowing

This section shows how topics and messages automatically replicate from the source to the shadow cluster.

  1. Create a topic on the source cluster:

    rpk --profile source topic create basic -p1 -r1

  2. Produce a message to the source cluster:

    echo "Hello, world" | rpk --profile source topic produce basic

  3. Wait a few seconds for replication, then list topics on the shadow cluster:

    rpk --profile shadow topic list

    You should see the basic topic appear on the shadow cluster.

  4. Consume messages from both clusters to verify data replication:

    rpk --profile source topic consume basic -n1
rpk --profile shadow topic consume basic -n1

    The same message appears on both clusters.

Explore Schema Registry shadowing

This section shows how Avro schemas registered on the source cluster replicate to the shadow cluster, allowing shadow-side message deserialization.

  1. Register an Avro schema on the source cluster:

    rpk --profile source registry schema create syslog-value --schema resources/syslog.avsc --type avro

  2. Create a topic for schema-encoded messages:

    rpk --profile source topic create syslog -p1 -r1

  3. Produce Avro-encoded syslog records to the source cluster:

    cat resources/syslogs.json | rpk --profile source topic produce syslog --schema-id=topic

  4. Wait a few seconds, then list schemas on both clusters:

    rpk --profile source registry subject list
rpk --profile shadow registry subject list

    The syslog-value schema appears on both clusters.

  5. Consume decoded messages from the shadow cluster using the replicated schema:

    rpk --profile shadow topic consume syslog -n1 --use-schema-registry -f'%v\n' | jq

    The shadow cluster can deserialize messages using the replicated schema.

Explore consumer group failover

This section shows how consumer group offsets replicate to the shadow cluster, allowing consumers to resume from their exact position after a failover.

  1. Create a topic on the source cluster:

    rpk --profile source topic create foo -p1 -r1

  2. Produce records and consume them on the source cluster:

    seq 0 2 | rpk --profile source topic produce foo
rpk --profile source topic consume foo -n3 -g consumer-group-foo

    This creates a consumer group and processes the messages.

  3. View consumer group offsets on both clusters:

    rpk --profile source group describe consumer-group-foo
rpk --profile shadow group describe consumer-group-foo

    The consumer group and its offsets are replicated to the shadow cluster.

  4. Trigger a failover to the shadow cluster:

    rpk --profile shadow shadow failover shadow-link --all --no-confirm
rpk --profile shadow shadow status shadow-link

    This command promotes the shadow cluster to become writable.

  5. Produce more records and consume on the shadow cluster:

    seq 3 5 | rpk --profile shadow topic produce foo
rpk --profile shadow topic consume foo -n3 -g consumer-group-foo

    The consumer resumes from the last committed offset (3) and consumes the new messages.

Clean up

To stop port forwarding, terminate the background processes:

kill $(cat local-port-forward.pid)

To delete the kind cluster and all resources:

kind delete cluster --name redpanda-shadow

Suggested reading