Skip to main content
Version: 22.3

Sizing Use Cases

The following scenarios provide estimates and advice for sizing Redpanda clusters for different throughput and retention use cases in your data center and in the cloud. For details about sizing considerations, see Sizing Guidelines.

note

These use cases assume a happy path with known metrics and expected outputs, but many other factors can influence performance, such as batch size and other sources of network traffic.

Low throughput, short retention

MetricValue
Producer throughput75 MB/sec (600 Mbps)
Producer rate300 messages per second
Consumer throughput75 MB/sec (600 Mbps)
Consumer rate300 messages per second
Data retention3 days
Average message size250 KB
Failure tolerance1 node

In this use case, despite the relatively low throughput of 150 MB/sec (producer plus consumer), it’s important to calculate the expected bandwidth utilization and to use a network testing tool like iPerf to verify that the bandwidth is available and sustainable. With a single topic with a replication factor of three, producing 75 MB/sec generates an additional 150 MB/sec of data transmitted over the network for replication, and it generates a further 75 MB/sec for the consumers.

The 150 MB/sec of bandwidth for replication is full duplex (where each byte sent by a broker is received by some other broker). The 75 MB/sec producer and consumer flows, however, are half-duplex, because the client endpoint in each case is outside of the cluster. Therefore, the intra-cluster bandwidth is 225 MB for incoming and outgoing flows:

  • 150 MB/sec of intra-cluster full duplex bandwidth
  • 75 MB/sec of ingress from producers
  • 75 MB/sec of egress to consumers

Three nodes satisfy Redpanda’s minimum deployment requirement (so Raft can form quorums) and also the single node failure tolerance. Divide the bandwidth total by the node count (3) to get the per-node bandwidth requirements. The throughput is not high enough to warrant any more than two cores and a single NVMe SSD disk. Be mindful of predicted growth of CPU and disk usage, and estimate when the cluster might need to scale up or scale out.

With an average producer throughput of 75 MB/sec and a replication factor of three, each node writes 254 GB of data each hour and 6.4 TB of data each day. For three days of data retention, each node needs at least 20 TB of storage.

note

This assumes that each node could be a leader or a follower, and there are a sufficient number of partitions for good distribution. A typical node is the leader for 1/Nth of the partitions in a cluster with N nodes and a follower for 2/Nths of the partitions. However, the per node bandwidth could vary if distribution is uneven. You may have an inexact distribution of load during Redpanda partition balancing or when the client library doesn't write to each partition evenly.

The following machine specifications provide a minimum for a bare metal cluster or its cloud-based equivalent.

Bare MetalAWSGCPAzure
Instance Type-m5.largen2-standard-2F2s_v2
Nodes3333
Cores2222
Memory4 GB8 GB8 GB4 GB
Instance Storage20 TB (NVMe)--16 GB (SSD)
Persistent Storage-20 TB (gb3)20 TB (Zonal SSD PD)20 TB (Standard SSD)
Network4 GbpsUp to 10 Gbps10 Gbps5 Gbps
Tiered StorageFalseFalseFalseFalse

Medium throughput, long retention

MetricValue
Producer throughput~500 MB/sec (~4,000 Mbps)
Producer rate2,000 messages per second
Consumer throughput~1,000 MB/sec (~8,000 Mbps)
Consumer rate4,000 messages per second
Data retention1 year
Average message size250 KB
Failure tolerance1 node

This use case requires a long duration of data retention (1 year). Producing an average of 500 MB/sec and consuming an average of 1,000 MB/sec equates to 2,500 MB/sec (20 Gbps) of network bandwidth for replication traffic. This is attainable but expensive with cloud providers, and these speeds are not as prevalent throughout a typical data center.

With at least one partition for each core, the 500 MB/sec of data from producers is evenly distributed between the nodes. For example, with three nodes, each node receives approximately 167 MB/sec. However, that bandwidth value increases with data replication.

Producer MB/secConsumer MB/secAvg. Replication FactorNodesWrites per node MB/secReads per node MB/sec
5001,50033500/3 * 3 = 5001500/3 = 500
5001,50035500/5 * 3 = 3001500/5 = 300
5001,50037500/7 * 3 = 2151500/7 = 215
5001,50057500/7 * 5 = 3581500/7 = 215

The additional 500 MB/sec for consumer throughput is for Tiered Storage and the bandwidth required to archive log segments to object storage. When Tiered Storage is enabled on a topic, it essentially adds another consumer’s worth of bandwidth on the network.

Producing an average of 500 MB every second equates to over 40 TB of data each day. With a retention period of one year, the cluster needs over 14 PB of storage. This is more than can be squeezed into three nodes, so you’re forced to overprovision just to handle the data retention requirements. This isn’t an issue in the cloud with high-performance persistent storage options (like io2 in AWS, Extreme PD in GCP, and Ultra SSD in Azure), but at this scale, it’s very expensive.

To balance the available local disk, consider exactly how many reads can be serviced from local storage. Different instance types or locally-attached NVMe SSD disks provide different amounts of local storage, and therefore different amounts of available data without going back to the cloud.

A topic with Tiered Storage enabled can write data to faster local storage managed by local retention settings, and at the same time, it can write data to object storagage managed by different retention settings, or left to grow indefinitely. Consumers that generally keep up with producers stream from local storage, but at this velocity that window of opportunity is narrower. The object store is a backup if a consumer needs to read from an older offset.

Bare MetalAWSGCPAzure
Instance Type-i3en.6xlargen2-standard-32F48s_v2
Nodes3333
Cores24243248
Memory192 GB192 GB128 GB96 GB
Instance Storage30 TB (NVMe)15 TB (NVM3)9 TB (SSD)384 TB (SSD)
Persistent Storage---20 TB (Standard SSD)
Available Local Retention17 hrs8 hrs5 hrs9 days
Network25 Gbps25 Gbps32 Gbps21 Gbps
Tiered StorageTrueTrueTrueTrue

High throughput, long retention

MetricValue
Producer throughput1,000 MB/sec (8,000 Mbps)
Producer rate4,000 messages per second
Consumer throughput2,000 MB/sec (16,000 Mbps)
Consumer rate8,000 messages per second
Data retention1 year
Average message size250 KB
Failure tolerance2 nodes

This use case has many topics, hundreds of partitions, and a high throughput. The combined producer and replication data equates to 8 Gbps of network traffic, plus 16 Gbps for the consumers and 8 Gbps for Tiered Storage. In total, that’s at least 32 Gbps of network bandwidth required to sustain this level of throughput. Writing at 1,000 MB/sec is near the upper limit of what a single NVMe disk can sustain.

At this scale, you get significant performance gains by distributing the writes over many cores and disks to better leverage Redpanda’s thread-per-core model. For example, given five nodes with 24 cores each, start with at least one partition for each core (120 partitions in total) and scale up. Redpanda generates over 3 TB of writes each hour and over 80 TB each day. Local storage is going to fill up quickly, and the window of opportunity for consumers to read from local storage is going to be shorter than in the other scenarios. In this use case, Tiered Storage is essential.

Bare MetalAWSGCPAzure
Instance Type-i3en.12xlargen2-standard-48F48s_v2
Nodes5555
Cores24484848
Memory192 GB384 GB192 GB96 GB
Instance Storage30 TB (NVMe)30 TB (NVM3)9 TB (SSD)384 TB (SSD)
Persistent Storage---30 TB (Ultra SSD)
Available Local Retention14 hrs7 hrs4 hrs7 days
Network25 Gbps25 Gbps32 Gbps21 Gbps
Tiered StorageTrueTrueTrueTrue

What do you like about this doc?




Optional: Share your email address if we can contact you about your feedback.

Let us know what we do well: