Manage the throughput of Kafka traffic at the cluster level, with configurable properties that limit and protect the use of disk and network resources for individual nodes and for an entire cluster. Set node-wide throughput limits for Kafka API traffic.
Node-wide throughput limits
The network bandwidth and disk utilization of cluster nodes may be overloaded by clients that produce or consume throughput without limits. To prevent resource overloading caused by unconstrained throughput, Redpanda provides runtime-configurable properties that limit and balance throughput of Kafka API traffic.
To manage the volume of traffic going through a node, Redpanda implements throughput quotas on the ingress and egress sides of every node. The throughput quota accounts for all Kafka API traffic going in or out of a node, with the value of quota representing the allowed rate of data passing through in one direction. When a connection is in breach of the quota, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level, and it implements that delay before handling Kafka API requests. To control the quotas, Redpanda provides configurable rate limits for total ingress and egress traffic through a node.
With Redpanda's thread-per-core model, the Kafka API traffic to and from a client connection is processed by a single core (shard). In order to manage throughput quotas efficiently, node quotas are distributed between shards, and each per-shard quota is in turn shared by all connections served by the shard. Splitting node quota optimally between shards is done behind the scenes by the quota balancer component.
To distribute the node throughput quota, the balancer periodically monitors the throughput rate of a node's shards, and it distributes more quota to the shards that can make better use of it than the others. Each shard has a minimum throughput quota value, which is configurable both as a percentage of the default quota and as an absolute rate limit.
The properties for node-wide throughput quota balancing are configured at the cluster level, for all nodes in a cluster:
|kafka_throughput_limit_node_in_bps||A node's total throughput limit for ingress Kafka traffic.|
|kafka_throughput_limit_node_out_bps||A node's total throughput limit for egress Kafka traffic.|
|kafka_quota_balancer_node_period_ms||The period at which the quota balancer runs to balance throughput quota between a node's shards.|
|kafka_quota_balancer_min_shard_throughput_ratio||The lowest value of the throughput quota a shard can get in the process of quota balancing, expressed as a ratio of the default shard quota. If set as |
|kafka_quota_balancer_min_shard_throughput_bps||The lowest value of the throughput quota a shard can get in the process of quota balancing, in bytes per second. If set as |
|max_kafka_throttle_delay_ms||The maximum delay inserted in the data path of Kafka API requests to throttle them down. Configuring this to be less than the Kafka client timeout can ensure that the delay that's inserted won't be long enough to cause a client timeout by itself.|
|kafka_quota_balancer_window_ms||The time window the balancer uses to average the current throughput measurement.|
- By default, both
kafka_throughput_limit_node_out_bpsare disabled, no throughput limits are applied. You must manually set them to enable quota balancing with throughput limits.
kafka_quota_balancer_min_shard_throughput_bpsdoesn't override the
kafka_throughput_limit_node_out_bpslimit settings. Consequently, the value of
kafka_throughput_limit_node_out_bpscan result in lesser throughput than