Cloud Topics in Kubernetes
Cloud Topics use object storage as the primary data tier to optimize for latency-tolerant, high-throughput workloads.
Because the data lives in object storage, Cloud Topics also support recovery after total cluster loss: a whole cluster restore brings the topics back in cloud storage mode with their data.
How Cloud Topics work
With standard Redpanda topics, data is replicated across brokers using Raft consensus and stored locally on each replica. Cloud Topics change this model: data is acknowledged only after it is uploaded to object storage, making object storage the source of truth for both replication and consumption.
The end-to-end latency experienced when using Cloud Topics can range from 500 ms to as high as a few seconds with different object stores. Lower latencies may be achievable in certain environments, but Cloud Topics is optimized for throughput rather than low latency or tightly constrained tail latency. This latency profile is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective.
With Cloud Topics, data from the client is not acknowledged until it is uploaded to object storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. You should generally expect end-to-end latencies of 1-2 seconds with public cloud stores.
Storage modes
Redpanda supports multiple storage modes that you can set at the cluster or topic level using the redpanda.storage.mode property:
| Mode | Behavior |
|---|---|
|
Follows legacy behavior. Topics inherit cluster-level remote read/write settings. |
|
Data is stored only on local disk. No remote storage is used. |
|
Data is written locally and offloaded to object storage asynchronously using Tiered Storage. |
|
Data is managed primarily in object storage. Local storage acts as a cache. |
Ideal use cases
Cloud Topics are best suited for latency-tolerant workloads, including:
-
Observability and logging streams
-
Offline analytics pipelines
-
AI/ML training data ingestion
-
Development and staging environments with flexible latency requirements
Limitations
-
Shadow links do not currently support Cloud Topics.
-
Once created, a Cloud Topic cannot be converted back to a standard Redpanda topic that uses local or Tiered Storage. Conversely, existing topics created as local or Tiered Storage topics cannot be converted to Cloud Topics.
-
Higher produce latency compared to standard topics (expect 1-2 seconds with public cloud stores).
Configure Cloud Topics on Kubernetes to optimize for latency-tolerant, high-throughput workloads using object storage as the primary data tier.
Recover Cloud Topics onto a new Kubernetes cluster after total cluster loss, with topics returning in cloud storage mode and no manual data migration.
