Manage Throughput

Manage the throughput of Kafka traffic at the cluster level, with configurable properties that limit and protect the use of disk and network resources for individual nodes and for an entire cluster. Set node-wide throughput limits for Kafka API traffic.

Node-wide throughput limits

The network bandwidth and disk utilization of cluster nodes may be overloaded by clients that produce or consume throughput without limits. To prevent resource overloading caused by unconstrained throughput, Redpanda provides runtime-configurable properties that limit and balance throughput of Kafka API traffic.

To manage the volume of traffic going through a node, Redpanda implements throughput quotas on the ingress and egress sides of every node. The throughput quota accounts for all Kafka API traffic going in or out of a node, with the value of quota representing the allowed rate of data passing through in one direction. When a connection is in breach of the quota, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level, and it implements that delay before handling Kafka API requests. To control the quotas, Redpanda provides configurable rate limits for total ingress and egress traffic through a node.

With Redpanda's thread-per-core model, the Kafka API traffic to and from a client connection is processed by a single core (shard). In order to manage throughput quotas efficiently, node quotas are distributed between shards, and each per-shard quota is in turn shared by all connections served by the shard. Splitting node quota optimally between shards is done behind the scenes by the quota balancer component.

To distribute the node throughput quota, the balancer periodically monitors the throughput rate of a node's shards, and it distributes more quota to the shards that can make better use of it than the others. Each shard has a minimum throughput quota value, which is configurable both as a percentage of the default quota and as an absolute rate limit.

The properties for node-wide throughput quota balancing are configured at the cluster level, for all nodes in a cluster:

Property	Description
kafka_throughput_limit_node_in_bps	A node's total throughput limit for ingress Kafka traffic.
kafka_throughput_limit_node_out_bps	A node's total throughput limit for egress Kafka traffic.
kafka_quota_balancer_node_period_ms	The period at which the quota balancer runs to balance throughput quota between a node's shards.
kafka_quota_balancer_min_shard_throughput_ratio	The lowest value of the throughput quota a shard can get in the process of quota balancing, expressed as a ratio of the default shard quota. If set as 0, there is no minimum, and if set as 1, no quota can be taken away by the balancer.
kafka_quota_balancer_min_shard_throughput_bps	The lowest value of the throughput quota a shard can get in the process of quota balancing, in bytes per second. If set as 0, there is no minimum.
max_kafka_throttle_delay_ms	The maximum delay inserted in the data path of Kafka API requests to throttle them down. Configuring this to be less than the Kafka client timeout can ensure that the delay that's inserted won't be long enough to cause a client timeout by itself.
kafka_quota_balancer_window_ms	The time window the balancer uses to average the current throughput measurement.

notes

• By default, both kafka_throughput_limit_node_in_bps and kafka_throughput_limit_node_out_bps are disabled, no throughput limits are applied. You must manually set them to enable quota balancing with throughput limits.
• kafka_quota_balancer_min_shard_throughput_bps doesn't override the kafka_throughput_limit_node_in_bps and kafka_throughput_limit_node_out_bps limit settings. Consequently, the value of kafka_throughput_limit_node_in_bps or kafka_throughput_limit_node_out_bps can result in lesser throughput than kafka_quota_balancer_min_shard_throughput_bps.