Remote Read Replicas

Beta

A remote read replica topic is a read-only topic that mirrors a topic on a different cluster. You can create a separate remote cluster just for consumers of this topic and populate its topics from object storage. A read-only topic on a remote cluster can serve any consumer, without increasing the load on the source cluster. Because these read-only topics access data directly from object storage, there’s no impact to the performance of the cluster.

Redpanda Cloud supports remote read replicas with ephemeral BYOC clusters (not with a customer-managed VPC). Ephemeral clusters are temporary clusters created to handle specific, transient workloads or tasks. After processing, the cluster is terminated to free up resources and reduce costs.

Prerequisites

  • A BYOC source cluster in Ready state.

  • A BYOC reader cluster in Ready state. This separate reader cluster must exist in the same Redpanda organization and the same cloud provider account and region as the source cluster.

Configure remote read replicas

You can add or remove reader clusters to a source cluster in Redpanda Cloud with the Cloud API. For information on accessing the Cloud API, see Redpanda Cloud API Authentication.

  1. To update your source cluster to add one or more reader cluster IDs, make a PATCH /v1beta2/clusters/\{cluster.id} request. The full list of clusters is expected on every call. If an ID is removed from the list, it will be removed as a reader cluster.

    export SOURCE_CLUSTER_ID=.......
export READER_CLUSTER_ID=.......

curl -X PATCH $API_HOST/v1beta2/clusters/$SOURCE_CLUSTER_ID \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_TOKEN" \
-d @- << EOF
{
  "read_replica_cluster_ids": ["$READER_CLUSTER_ID"]
}
EOF

  2. To create a remote read replica topic, run:

    rpk topic create my-topic -c redpanda.remote.readreplica=redpanda-cloud-storage-${SOURCE_CLUSTER_ID}

  3. Optional: To see the list of reader clusters on a given source cluster, make a GET /v1beta2/clusters/{id} request:

    export SOURCE_CLUSTER_ID=.......

curl -X GET $API_HOST/v1beta2/clusters/$SOURCE_CLUSTER_ID \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_TOKEN"

  4. Optional: For remote read replicas reading from a live topic (that is, a topic that’s being actively written to by a source cluster), it may be advantageous to control how often segments are flushed to object storage. By default, this is set to 60 minutes. To tune cloud_storage_segment_max_upload_interval_sec on the source cluster, contact Redpanda support. (For cold topics, where segments are closed and older than 60 minutes, this configuration is unnecessary: the data is already uploaded to object storage.)

A source cluster cannot be deleted if it has remote read replica topics. When you delete a reader cluster, that cluster’s ID is removed from any existing source cluster read_replica_cluster_ids lists.