24.1 23.3 23.2

Manage Throughput

Redpanda supports applying throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. The purpose of this is to prevent unbounded network and disk usage of the broker by clients. Broker-wide limits apply to all clients connected to the broker and restrict total traffic on the broker. Client limits apply to a set of clients defined by their client_id and help prevent a set of clients from starving other clients using the same broker.

Throughput throttling enforcement

Throughput limits are enforced by applying backpressure to clients. When a connection is in breach of the throughput limit, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level. Redpanda starts by adding a throttle_time_ms field to responses. If that isn’t honored, delays are inserted on the connection’s next read operation. The throttling delay may not exceed the limit set by max_kafka_throttle_delay_ms.

Broker-wide throughput limits

Broker-wide throughput limits account for all Kafka API traffic going in to or out of the broker. The limit values represent the allowed rate of data in bytes per second passing through in each direction. Redpanda also provides administrators the ability to exclude clients from throughput throttling and to fine-tune which Kafka request types are subject to throttling limits.

Broker-wide throughput limit properties

The properties for broker-wide throughput quota balancing are configured at the cluster level, for all brokers in a cluster:

Property Description

kafka_throughput_limit_node_in_bps

A broker’s total throughput limit for ingress Kafka traffic.

kafka_throughput_limit_node_out_bps

A broker’s total throughput limit for egress Kafka traffic.

kafka_throughput_control

List of clients for whom broker-wide limits do not apply

kafka_throughput_controlled_api_keys

List of Kafka request types subject to broker-wide throughput limits; defaults to produce and fetch.

max_kafka_throttle_delay_ms

The maximum delay inserted in the data path of Kafka API requests to throttle them down. Configuring this to be less than the Kafka client timeout can ensure that the delay that’s inserted won’t be long enough to cause a client timeout by itself.

  • By default, both kafka_throughput_limit_node_in_bps and kafka_throughput_limit_node_out_bps are disabled, and no throughput limits are applied. You must manually set them to enable throughput throttling.

Client throughput limits

Similar to Broker-wide throughput limits but for clients, Redpanda provides configurable throughput quotas that apply to an individual client or a group of clients.

The client throughput quotas limit the rates within each shard (logical CPU core) of a Redpanda broker’s node. The quotas are neither shared nor balanced between shards and brokers.

Individual client throughput limit

To set throughput quota for a single client, configure the cluster properties target_quota_byte_rate and target_fetch_quota_byte_rate.

  • The target_quota_byte_rate property applies to a producer client that isn’t a member of a client group configured by kafka_client_group_byte_rate_quota. It sets the maximum throughput quota of a client sending to a Redpanda broker.

  • The target_fetch_quota_byte_rate property applies to a consumer client that isn’t a member of a client group configured by kafka_client_group_fetch_byte_rate_quota. It sets the maximum throughput quota of a client fetching from a Redpanda broker.

The values of both target_quota_byte_rate and target_fetch_quota_byte_rate are throughput rate limits within a shard, in bytes per second.

Group of clients throughput limit

To set throughput quota for a group of clients, configure the cluster properties kafka_client_group_byte_rate_quota and kafka_client_group_fetch_byte_rate_quota.

  • The kafka_client_group_byte_rate_quota property applies to producer clients. It sets a maximum throughput quota of traffic sent to Redpanda from each producer in the group named by the property.

  • The kafka_client_group_fetch_byte_rate_quota property applies to consumer clients. It sets a maximum throughput quota of traffic fetched from Redpanda by each consumer in the group named by the property.

Both kafka_client_group_byte_rate_quota and kafka_client_group_fetch_byte_rate_quota have the configuration value fields:

  • group_name: a name for the group of clients.

  • clients_prefix: a client belonging to the group has this prefix in its client_id property.

  • quota: the maximum throughput rate of each client in the group in bytes per second.

For kafka_client_group_fetch_byte_rate_quota, a group of consumer clients is not a Kafka consumer group, and the group_name field is not a Kafka consumer group ID. Instead, it is the set of clients fetching from a Redpanda broker that are configured by the property to be throughput limited.

An example configuration of kafka_client_group_byte_rate_quota for two groups of producers:

"kafka_client_group_byte_rate_quota": [
  {
    "group_name": "group_1",
    "clients_prefix": "producer_group_alone_producer",
    "quota": 10240
  },
  {
    "group_name": "group_2",
    "clients_prefix": "producer_group_multiple",
    "quota": 20480
  },
]