Streaming

Restore Cloud Topics with Whole Cluster Restore

You can recover Cloud Topics onto a new cluster after the loss of an entire Redpanda cluster. Because object storage is the source of truth for Cloud Topics, a Whole Cluster Restore brings the topics back in cloud storage mode, with their data and offsets, ready to serve producers and consumers again.

After reading this page, you will be able to:

  • Deploy a target cluster that restores Cloud Topics from object storage on bootstrap

  • Verify that restored topics retain cloud storage mode and their data

  • Identify the recovery point for Cloud Topics data

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

How Whole Cluster Restore works with Cloud Topics

Cloud Topics acknowledge writes only after the data is uploaded to object storage, so the bucket already holds both the topic data and the cluster metadata that Whole Cluster Restore needs. When you bootstrap a new (target) cluster against the same bucket with restore enabled, Redpanda:

  • Recreates the topics from the most recent cluster metadata upload.

  • Restores Cloud Topics in cloud storage mode (redpanda.storage.mode=cloud). Topics do not fall back to Tiered Storage or local storage.

  • Restores users, ACLs, schemas, consumer offsets, and cluster configuration, as described in Whole Cluster Restore.

Topics without per-topic archival to object storage are restored empty, with their definitions but no data. This does not apply to Cloud Topics (redpanda.storage.mode=cloud), whose data always resides in object storage and is fully restored.

Restored Cloud Topics are immediately writable. New records continue from the next offset after the restored high watermark.

Recovery point

Whole Cluster Restore recovers each Cloud Topic partition up to the point covered by the most recent topic metadata in object storage. Records that were acknowledged shortly before the cluster loss might not yet be reflected in that metadata and are not restored.

For planned scenarios, such as migrations or recovery drills, stop producers and wait for metadata uploads to complete before removing the source cluster. This results in a restore with no data loss. To make uploads more frequent, see Manage source metadata uploads.

Prerequisites

You must have the following:

  • Redpanda v26.1 or later on both the source and target clusters. Cloud Topics require v26.1 or later.

  • Cloud Topics enabled on the source cluster, with cluster metadata uploads enabled (enable_cluster_metadata_upload_loop is enabled by default).

  • The object storage bucket or container used by the source cluster. The restore reads everything from this bucket, so it must survive the cluster loss.

  • A valid Redpanda Enterprise license for the target cluster.

The limitations in Whole Cluster Restore also apply to clusters with Cloud Topics.

Start a target cluster with restore enabled

Here we describe how to deploy a new, empty cluster configured with the same object storage settings as the failed source cluster, with Cloud Topics enabled, and with restore on bootstrap enabled.

Before you bootstrap the target cluster against the bucket, make sure that the failed source cluster is fully destroyed, including any persistent volumes. If brokers from the previous cluster are still running against the same bucket, the storage subsystems of the two clusters may interfere with each other.

  • Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    storage:
      tiered:
        <object-storage-settings>
    config:
      cluster:
        cloud_topics_enabled: true
        cloud_storage_attempt_cluster_restore_on_bootstrap: true
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

  • --values

  • --set

cluster-restore.yaml
storage:
  tiered:
    <object-storage-settings>
config:
  cluster:
    cloud_topics_enabled: true
    cloud_storage_attempt_cluster_restore_on_bootstrap: true
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
--values cluster-restore.yaml
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set storage.tiered.<object-storage-settings> \
  --set config.cluster.cloud_topics_enabled=true \
  --set config.cluster.cloud_storage_attempt_cluster_restore_on_bootstrap=true

  • storage.tiered: Configure the target cluster with the same object storage settings as the failed source cluster, including the same bucket or container.

  • config.cluster.cloud_topics_enabled: Enable Cloud Topics on the target cluster so that restored topics can be served in cloud storage mode.

  • config.cluster.cloud_storage_attempt_cluster_restore_on_bootstrap: Run the restore automatically when the new cluster bootstraps. No manual restore command is required.

The restore runs while the cluster starts. Cluster size does not need to match the source cluster, but the target cluster must be empty of user-managed data.

Verify that the restore is complete

  1. Run the following command until it returns inactive:

    rpk cluster storage restore status

  2. Check if a rolling restart is required:

    rpk cluster config status

    Example output when a restart is required:

    NODE  CONFIG-VERSION  NEEDS-RESTART  INVALID  UNKNOWN
1     4               true           []       []

  3. If a restart is required, perform a rolling restart.

  4. After the restore completes, set cloud_storage_attempt_cluster_restore_on_bootstrap back to false so that future bootstraps of the same deployment do not attempt another restore. Update the value in your Redpanda resource or Helm values, then perform a rolling restart if rpk cluster config status reports one is needed.

Verify restored Cloud Topics

  1. Confirm that the topic is restored in cloud storage mode:

    rpk topic describe <topic-name> -c

    The output includes redpanda.storage.mode with the value cloud:

    redpanda.storage.mode    cloud    DYNAMIC_TOPIC_CONFIG

    If the topic inherited cloud storage mode from the cluster default (default_redpanda_storage_mode), the source column shows DEFAULT_CONFIG instead of DYNAMIC_TOPIC_CONFIG.

  2. Confirm that the partitions contain the restored data:

    rpk topic describe <topic-name> -p

    The high watermark of each partition reflects the restored records.

  3. Confirm that the topic accepts new records:

    echo "restore-probe" | rpk topic produce <topic-name>

    The record is produced at the next offset after the restored high watermark.

When verification succeeds, redirect your application workload to the new cluster. Make sure to update your application code to use the new addresses of your brokers.

Next steps

Suggested reading