Create Remote Read Replicas

Beta

A remote read replica topic is a read-only topic that mirrors a topic on a different cluster. You can create a separate remote cluster just for consumers of this topic and populate its topics from object storage. A read-only topic on a remote cluster can serve any consumer, without increasing the load on the source cluster. Because these read-only topics access data directly from object storage, there’s no impact to the performance of the cluster.

Redpanda Cloud supports remote read replicas with ephemeral BYOC clusters. Ephemeral clusters are temporary clusters created to handle specific, transient workloads or tasks. After processing, the cluster is terminated to free up resources and reduce costs.

Prerequisites

  • A BYOC source cluster in Ready state.

  • A BYOC reader cluster in Ready state. This separate reader cluster must exist in the same Redpanda organization and the same cloud provider account and region as the source cluster.

Customer-managed VPC: Grant storage permissions

This prerequisite only applies to customer-managed VPC deployments. Skip this step if you’re enabling remote read replicas on standard BYOC clusters.

  • GCP

  • AWS

To grant additional permissions to the cloud storage manager of the reader cluster, run:

gcloud storage buckets add-iam-policy-binding \
    gs://<source cluster tiered storage bucket name> \
    --member="serviceAccount:<reader cluster redpanda_cluster_service_account email>" \
    --role="roles/storage.objectViewer"

To grant additional permissions to the cloud storage manager of the reader cluster, set the source_cluster_bucket_names and reader_cluster_id variables in cloud-examples. This should be done in the Terraform of the reader cluster.

Configure remote read replica

Add or remove reader clusters to a source cluster in Redpanda Cloud with the Cloud API. For information on accessing the Cloud API, see Redpanda Cloud API Authentication.

  1. To update your source cluster to add one or more reader cluster IDs, make a PATCH /v1beta2/clusters/\{cluster.id} request. The full list of clusters is expected on every call. If an ID is removed from the list, it is removed as a reader cluster.

    export SOURCE_CLUSTER_ID=.......
export READER_CLUSTER_ID=.......

curl -X PATCH $API_HOST/v1beta2/clusters/$SOURCE_CLUSTER_ID \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_TOKEN" \
-d @- << EOF
{
  "read_replica_cluster_ids": ["$READER_CLUSTER_ID"]
}
EOF

  2. Optional: To see the list of reader clusters on a given source cluster, make a GET /v1beta2/clusters/{id} request:

    export SOURCE_CLUSTER_ID=.......

curl -X GET $API_HOST/v1beta2/clusters/$SOURCE_CLUSTER_ID \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_TOKEN"

A source cluster cannot be deleted if it has remote read replica topics. When you delete a reader cluster, that cluster’s ID is removed from any existing source cluster read_replica_cluster_ids lists.

Create remote read replica

To create a remote read replica topic, run:

rpk topic create my-topic -c redpanda.remote.readreplica=<source cluster tiered storage bucket name>

For standard BYOC clusters the source cluster tiered storage bucket name follows the pattern: redpanda-cloud-storage-${SOURCE_CLUSTER_ID}

Optional: Tune for live topics

For remote read replicas reading from a live topic (that is, a topic that’s being actively written to by a source cluster), it may be advantageous to control how often segments are flushed to object storage. By default, this is set to 60 minutes. To tune cloud_storage_segment_max_upload_interval_sec on the source cluster, contact Redpanda support. (For cold topics, where segments are closed and older than 60 minutes, this configuration is unnecessary: the data is already uploaded to object storage.)