Enable Rack Awareness
Rack awareness allows you to distribute replicas of the same partition across different racks to minimize data loss in the event of a rack failure. A rack is a failure zone that has one or more Redpanda brokers assigned to it.
When you create a topic, you specify the number of partitions for the topic and the number of partition replicas. By default, Redpanda determines where to place the replicas on the cluster such that each replica is on a different broker, if possible.
By defining different racks for a Redpanda cluster, you can specify a preference for the way partition replicas are assigned to brokers. When Redpanda places partition replicas, it takes into account whether a replica has already been placed on a broker in a particular rack. If so, Redpanda chooses a broker in a different rack. This way, partition replicas are distributed across different failure zones, which provides a measure of fault tolerance in the event that a broker or an entire rack becomes unavailable.
When rack awareness is enabled, Redpanda places replicas according to these criteria:
- Number of racks vs. replicas - If the cluster has more racks than the number of replicas, each replica is placed on a broker in a unique rack. If the cluster has fewer racks than the number of replicas, some replicas are placed on brokers in the same rack.
- Number of available CPU cores - Brokers with more available CPU cores are chosen over brokers with fewer available CPU cores.
- Broker utilization - Brokers with fewer partitions are chosen over brokers with more partitions.
Setting up rack awareness
The instructions in this section are based on a cluster with five brokers and three failure zones. The failure zones, or racks, are identified as A, B, and C. The brokers are assigned to the three racks as shown:
To set up rack awareness for this cluster:
- Open a terminal window and log in to broker 1 on the cluster.
rpk cluster config edit.
- In the text editor window that opens, set
- Save your change and quit the editor.
- Go to the
- Open the
redpanda.yamlfile for editing.
- In the
redpandasection, add the
rackproperty and set the value to A as shown:
- Log in to the remaining brokers and edit their
redpanda.yamlfiles. For broker 2, set
A. For broker 3, set
B. For brokers 4 and 5, set
- Restart each broker in the cluster for the rack assignments to take effect.
For high availability, Redpanda recommends adding one or more brokers from each zone to the
Use rack awareness with Continuous Data Balancing to continually maintain the configured replication level, even after a rack failure. For a given partition, Redpanda tries to move excess replicas from racks that have more than one replica to racks that have no replicas.