23.2 23.1 22.3 22.2 22.1 21.11

Cluster Balancing

When a topic is created, Redpanda evenly distributes its partitions by sequentially allocating them to the node with the least number of partitions. By default, Redpanda provides leadership balancing and partition rebalancing when nodes are added or decommissioned.

With an Enterprise license, you can additionally enable Continuous Data Balancing to continuously monitor node and rack availability and disk usage. This enables self-healing clusters that dynamically balance partitions. It also continuously maintains adherence to rack-aware replica placement policy and self heals after rack (or availability zone) failure or replacement. See Continuous Data Balancing.

Cluster balancing protects you from unbalanced systems that saturate resources on one or more nodes. This can affect throughput and latency. Furthermore, a cluster with replicas on a down node risks availability loss if more nodes fail, and a cluster that keeps losing nodes without healing eventually risks data loss.

Leadership balancing

Automatic leadership balancing improves cluster performance by transferring leadership for a node’s partitions to other replicas. Automatic leadership balancing changes where data is read to and written to first. It doesn’t move any data.

Leadership balancing is enabled by default with the enable_leader_balancer property.

Partition balancing

Partition balancing moves data from hot-loaded nodes to least-loaded nodes to ensure steady-state performance. It ensures predictable and even performance across all nodes in a cluster. Partition balancing is invoked periodically, determined by the partition_autobalancing_tick_interval_ms property. Default is 30 seconds.

By default, Redpanda rebalances partition distribution when nodes are added or decommissioned. Continuous Data Balancing additionally rebalances partitions when nodes become unavailable or when disk space usage exceeds a threshold.

  • Monitoring unavailable nodes lets Redpanda self-heal clusters by moving partitions from a failed node to a healthy node.

  • Monitoring low disk space lets Redpanda distribute partitions across nodes with enough disk space. If free disk space reaches a critically low level, Redpanda blocks clients from producing. For information about the disk space threshold and alert, see Handle full disks.

Partition balancing settings

Select your partition balancing setting with the partition_autobalancing_mode property.

Setting Description

node_add

Partition balancing happens when nodes are added.

This is the default setting.

continuous

In this mode, Redpanda continuously monitors the cluster for node failures and high disk usage. It uses this information to automatically redistribute partitions across the cluster to maintain optimal performance and availability. It also monitors rack availability after failures, and for a given partition, it tries to move excess replicas from racks that have more than one replica to racks where there are none. See Continuous Data Balancing.

This option requires an Enterprise license.

off

All partition balancing from Redpanda is turned off.

This option is not recommended for production clusters. Only set to off if you need to move partitions manually.