Leader Pinning
Produce requests that write data to Redpanda topics go through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), leader pinning ensures that a topic’s partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency.
If consumers are located in the same preferred region or AZ for leader pinning, and you have not set up follower fetching, leader pinning can also help reduce networking costs on consume requests.
Prerequisites
This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales. If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply. |
Before you can enable leader pinning, you must configure rack awareness on the cluster. If the enable_rack_awareness
cluster configuration property is set to false
, leader pinning is disabled across the cluster.
Configure leader pinning
You can use both a topic configuration property and a cluster configuration property to configure leader pinning.
You can set the topic configuration property for individual topics only, or set the cluster-wide configuration property that will enable leader pinning by default for all topics. You can also use a combination in which a default setting applies across the cluster, and you toggle the setting on or off for specific topics.
This configuration is based on the following scenario: you have Redpanda deployed in a multi-AZ or multi-region cluster, and you have configured each broker so that the rack
configuration property contains racks corresponding to the AZs:
-
Set the topic configuration property
redpanda.leaders.preference
. The property accepts the following string values:-
none
: Opt out the topic from leader pinning. -
racks:<rack1>[,<rack2>,…​]
: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more racks, and you can list the racks in any order. Spaces in the list are ignored, for example:racks:rack1,rack2
andracks: rack1, rack2
are equivalent. You cannot specify empty racks, for example:racks: rack1,,rack2
. If you specify multiple racks, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.To find the rack identifier, run
rpk cluster info
.
This property inherits the default value from the cluster property
default_leaders_preference
. -
-
Set the cluster configuration property
default_leaders_preference
, which specifies the default leader pinning configuration for all topics that don’t haveredpanda.leaders.preference
explicitly set. It accepts values in the same format asredpanda.leaders.preference
. Default:none
This property also affects internal topics, such as
__consumer_offsets
and transaction coordinators. All offset tracking and transaction coordination requests get placed within the preferred regions or AZs for all clients, so you see end-to-end latency and networking cost benefits.
If there is more than one broker in the preferred AZ (or AZs), leader pinning distributes partition leaders uniformly across brokers in the AZ.
Leader pinning failover across availability zones
If there are three AZs: A, B, and C, and A becomes unavailable, the failover behavior is as follows:
-
A topic with "A" as the preferred leader AZ will have its partition leaders uniformly distributed across B and C.
-
A topic with "A,B" as the preferred leader AZs will have its partition leaders in B.
-
A topic with “B” as the preferred leader AZ will have its partition leaders in B as well.