Collapse navigation tree

Collapse

Cloud Topics on Kubernetes

Starting in v26.1, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a per-topic configuration running mixed latency workloads. While standard Redpanda topics that use local storage or Tiered Storage are ideal for latency-sensitive workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements.

Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to both replicate data and serve it to consumers. This eliminates over 90% of the cost of replicating data over network links in multi-AZ clusters.

After reading this page, you will be able to:

  • Configure object storage for Cloud Topics on Kubernetes

  • Enable and create Cloud Topics using the Redpanda Operator or Helm

  • Verify topic storage mode configuration

How Cloud Topics work

With standard Redpanda topics, data is replicated across brokers using Raft consensus and stored locally on each replica. Cloud Topics change this model: data is acknowledged only after it is uploaded to object storage, making object storage the source of truth for both replication and consumption.

The end-to-end latency experienced when using Cloud Topics can range from 500 ms to as high as a few seconds with different object stores. Lower latencies may be achievable in certain environments, but Cloud Topics is optimized for throughput rather than low latency or tightly constrained tail latency. This latency profile is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective.

With Cloud Topics, data from the client is not acknowledged until it is uploaded to object storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. You should generally expect end-to-end latencies of 1-2 seconds with public cloud stores.

Storage modes

Redpanda supports multiple storage modes that you can set at the cluster or topic level using the redpanda.storage.mode property:

Mode Behavior

unset

Follows legacy behavior. Topics inherit cluster-level remote read/write settings.

local

Data is stored only on local disk. No remote storage is used.

tiered

Data is written locally and offloaded to object storage asynchronously using Tiered Storage.

cloud

Data is managed primarily in object storage. Local storage acts as a cache.

Ideal use cases

Cloud Topics are best suited for latency-tolerant workloads, including:

  • Observability and logging streams

  • Offline analytics pipelines

  • AI/ML training data ingestion

  • Development and staging environments with flexible latency requirements

Limitations

  • Shadow links do not currently support Cloud Topics.

  • Once created, a Cloud Topic cannot be converted back to a standard Redpanda topic that uses local or Tiered Storage. Conversely, existing topics created as local or Tiered Storage topics cannot be converted to Cloud Topics.

  • Higher produce latency compared to standard topics (expect 1-2 seconds with public cloud stores).

Prerequisites

You must have the following:

  • kubectl: Ensure you have the kubectl command-line tool installed and configured to communicate with your cluster.

  • Redpanda: A Redpanda Operator and a Redpanda resource deployed in your Kubernetes cluster running Redpanda v26.1 or later.

  • Object storage: A configured object storage backend (Amazon S3, Google Cloud Storage, Azure Blob Storage, or an S3-compatible store such as MinIO).

  • Enterprise license: A valid Redpanda Enterprise license applied to the cluster.

    This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

    If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

    To check if you already have a license key applied to your cluster:

rpk cluster license info

Configure object storage

Cloud Topics use the same object storage configuration as Tiered Storage. If you have already configured object storage for Tiered Storage, you can skip this step and proceed to Enable Cloud Topics.

For detailed instructions including IAM role configuration, access key management, and encryption options, see Configure object storage in the Tiered Storage documentation.

The following examples show the minimum required configuration for each cloud provider.

Amazon S3

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    storage:
      tiered:
        credentialsSecretRef:
          accessKey:
            name: storage-secrets
            key: access-key
          secretKey:
            name: storage-secrets
            key: secret-key
        config:
          cloud_storage_enabled: true
          cloud_storage_credentials_source: config_file (1)
          cloud_storage_region: <region>
          cloud_storage_bucket: <bucket-name>
1 Use aws_instance_metadata instead if you are using an IAM role attached to your nodes.
cloud-storage.yaml
storage:
  tiered:
    credentialsSecretRef:
      accessKey:
        name: storage-secrets
        key: access-key
      secretKey:
        name: storage-secrets
        key: secret-key
    config:
      cloud_storage_enabled: true
      cloud_storage_credentials_source: config_file (1)
      cloud_storage_region: <region>
      cloud_storage_bucket: <bucket-name>
1 Use aws_instance_metadata instead if you are using an IAM role attached to your nodes.

For details on IAM roles, access keys, and AWS KMS encryption, see Amazon S3 in the Tiered Storage documentation.

Google Cloud Storage (GCS)

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    storage:
      tiered:
        credentialsSecretRef:
          accessKey:
            name: storage-secrets
            key: access-key
          secretKey:
            name: storage-secrets
            key: secret-key
        config:
          cloud_storage_enabled: true
          cloud_storage_credentials_source: config_file (1)
          cloud_storage_api_endpoint: storage.googleapis.com
          cloud_storage_region: <region>
          cloud_storage_bucket: <bucket-name>
1 Use gcp_instance_metadata instead if you are using a GCP service account attached to your nodes.
cloud-storage.yaml
storage:
  tiered:
    credentialsSecretRef:
      accessKey:
        name: storage-secrets
        key: access-key
      secretKey:
        name: storage-secrets
        key: secret-key
    config:
      cloud_storage_enabled: true
      cloud_storage_credentials_source: config_file (1)
      cloud_storage_api_endpoint: storage.googleapis.com
      cloud_storage_region: <region>
      cloud_storage_bucket: <bucket-name>
1 Use gcp_instance_metadata instead if you are using a GCP service account attached to your nodes.

For details on IAM roles, HMAC access keys, and customer-managed encryption keys, see Google Cloud Storage in the Tiered Storage documentation.

Azure Blob Storage

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    storage:
      tiered:
        credentialsSecretRef:
          accessKey:
            name: storage-secrets
            key: shared-key
        config:
          cloud_storage_enabled: true
          cloud_storage_credentials_source: config_file (1)
          cloud_storage_azure_storage_account: <account-name>
          cloud_storage_azure_container: <container-name>
1 Use azure_aks_oidc_federation instead if you are using an Azure managed identity. When using managed identities, omit credentialsSecretRef and configure workload identity annotations on the service account.
cloud-storage.yaml
storage:
  tiered:
    credentialsSecretRef:
      accessKey:
        name: storage-secrets
        key: shared-key
    config:
      cloud_storage_enabled: true
      cloud_storage_credentials_source: config_file (1)
      cloud_storage_azure_storage_account: <account-name>
      cloud_storage_azure_container: <container-name>
1 Use azure_aks_oidc_federation instead if you are using an Azure managed identity. When using managed identities, omit credentialsSecretRef and configure workload identity annotations on the service account.

For details on managed identities and account access keys, see Microsoft ABS/ADLS in the Tiered Storage documentation.

Enable Cloud Topics

To enable Cloud Topics, set the cloud_topics_enabled cluster property to true and set the default storage mode for all new topics to cloud.

  • Helm + Operator

  • Helm

  1. Add the following to your Redpanda custom resource to enable Cloud Topics and set the default storage mode:

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    storage:
      tiered:
        mountType: none
        credentialsSecretRef:
          accessKey:
            name: cloud-storage-creds
            key: access-key
          secretKey:
            name: cloud-storage-creds
            key: secret-key
        config:
          cloud_storage_enabled: true
          cloud_storage_region: <region>
          cloud_storage_bucket: <bucket-name>
    config:
      cluster:
        cloud_topics_enabled: true
        default_redpanda_storage_mode: cloud (1)
    1 Optional. Set to cloud to make all new topics Cloud Topics by default. Omit this to create Cloud Topics individually.

  2. Apply the configuration:

    kubectl apply -f redpanda-cluster.yaml -n <namespace>

  3. The Redpanda Operator reconciles the configuration. Wait for the cluster to be ready:

    kubectl rollout status statefulset redpanda -n <namespace> --watch

  4. Verify that Cloud Topics are enabled:

    kubectl exec -n <namespace> redpanda-0 -c redpanda -- \
  rpk cluster config get cloud_topics_enabled

    Expected output: true

  1. Add the following to your Helm values file:

    cloud-topics-values.yaml
    storage:
  tiered:
    mountType: none
    credentialsSecretRef:
      accessKey:
        name: cloud-storage-creds
        key: access-key
      secretKey:
        name: cloud-storage-creds
        key: secret-key
    config:
      cloud_storage_enabled: true
      cloud_storage_region: <region>
      cloud_storage_bucket: <bucket-name>
config:
  cluster:
    cloud_topics_enabled: true
    default_redpanda_storage_mode: cloud (1)
    1 Optional. Set to cloud to make all new topics Cloud Topics by default. Omit this to create Cloud Topics individually.

  2. Deploy or upgrade the Helm chart:

    helm upgrade --install redpanda redpanda/redpanda \
  -n <namespace> --create-namespace \
  -f cloud-topics-values.yaml

  3. Wait for the cluster to be ready:

    kubectl rollout status statefulset redpanda -n <namespace> --watch

  4. Verify that Cloud Topics are enabled:

    kubectl exec -n <namespace> redpanda-0 -c redpanda -- \
  rpk cluster config get cloud_topics_enabled

    Expected output: true

Create a Cloud Topic

You can create Cloud Topics either by setting the cluster default storage mode to cloud, or by configuring individual topics.

Create Cloud Topics by default

If you set default_redpanda_storage_mode to cloud in the cluster configuration, all new topics inherit the cloud storage mode automatically.

Verify the cluster default:

kubectl exec -n <namespace> redpanda-0 -c redpanda -- \
  rpk cluster config get default_redpanda_storage_mode

Expected output: cloud

Any new topic created without an explicit storage mode inherits this default:

cloud-topic.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Topic
metadata:
  name: my-cloud-topic
  namespace: <namespace>
spec:
  partitions: 3
  replicationFactor: 3
  cluster:
    clusterRef:
      name: redpanda
kubectl apply -f cloud-topic.yaml

Create individual Cloud Topics

If the cluster default storage mode is not set to cloud, you can create individual Cloud Topics by setting the redpanda.storage.mode property on the topic.

cloud-topic.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Topic
metadata:
  name: my-cloud-topic
  namespace: <namespace>
spec:
  partitions: 3
  replicationFactor: 3
  additionalConfig:
    redpanda.storage.mode: "cloud"
  cluster:
    clusterRef:
      name: redpanda
kubectl apply -f cloud-topic.yaml

Override the cluster default

Topic-level storage mode settings override the cluster default. For example, if the cluster default is cloud, you can create a topic that uses Tiered Storage instead:

tiered-topic.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Topic
metadata:
  name: my-tiered-topic
  namespace: <namespace>
spec:
  partitions: 3
  replicationFactor: 3
  additionalConfig:
    redpanda.storage.mode: "tiered"
  cluster:
    clusterRef:
      name: redpanda

Verify topic storage mode

To verify the storage mode of a topic, inspect its configuration through the Topic resource status:

kubectl get topic <topic-name> -n <namespace> -o jsonpath=\
  '{range .status.topicConfiguration[*]}{.name}={.value} ({.source}){"\n"}{end}' \
  | grep storage.mode

Expected output for a Cloud Topic:

storage.mode=cloud (DEFAULT_CONFIG)

Or, if explicitly set on the topic:

storage.mode=cloud (DYNAMIC_TOPIC_CONFIG)

The source field indicates whether the value was inherited from the cluster default (DEFAULT_CONFIG) or explicitly set on the topic (DYNAMIC_TOPIC_CONFIG).

Suggested reading