Manage Throughput

Redpanda supports applying throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. The purpose of this is to prevent unbounded network and disk usage of the broker by clients. Broker-wide limits apply to all clients connected to the broker and restrict total traffic on the broker. Client limits apply to a set of clients defined by their client_id and help prevent a set of clients from starving other clients using the same broker.

Throughput throttling enforcement

As of v24.2, Redpanda enforces all throughput limits per broker, including client throughput.

Throughput limits are enforced by applying backpressure to clients. When a connection is in breach of the throughput limit, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level. Redpanda starts by adding a throttle_time_ms field to responses. If that isn’t honored, delays are inserted on the connection’s next read operation. The throttling delay may not exceed the limit set by max_kafka_throttle_delay_ms.

Broker-wide throughput limits

Broker-wide throughput limits account for all Kafka API traffic going into or out of the broker, as data is produced to or consumed from a topic. The limit values represent the allowed rate of data in bytes per second passing through in each direction. Redpanda also provides administrators the ability to exclude clients from throughput throttling and to fine-tune which Kafka request types are subject to throttling limits.

Broker-wide throughput limit properties

The properties for broker-wide throughput quota balancing are configured at the cluster level, for all brokers in a cluster:

Cluster configuration property Description

kafka_throughput_limit_node_in_bps

A broker’s total throughput limit for ingress Kafka traffic.

kafka_throughput_limit_node_out_bps

A broker’s total throughput limit for egress Kafka traffic.

kafka_throughput_control

List of clients for whom broker-wide limits do not apply

kafka_throughput_controlled_api_keys

List of Kafka request types subject to broker-wide throughput limits; defaults to produce and fetch.

max_kafka_throttle_delay_ms

The maximum delay inserted in the data path of Kafka API requests to throttle them down. Configuring this to be less than the Kafka client timeout can ensure that the delay that’s inserted won’t be long enough to cause a client timeout by itself.

  • By default, both kafka_throughput_limit_node_in_bps and kafka_throughput_limit_node_out_bps are disabled, and no throughput limits are applied. You must manually set them to enable throughput throttling.

Client throughput limits

Redpanda provides configurable throughput quotas that apply to an individual client or a group of clients. You can apply a quota for an individual client based on an exact match with its client_id, or a group of clients based on IDs that start with a given prefix.

As of v24.2, client throughput quotas are compatible with the AlterClientQuotas and DescribeClientQuotas Kafka APIs, and are separate from quotas configured through cluster configuration in earlier Redpanda versions. The client throughput quotas no longer apply on a per-shard basis, and now limit the rates across a Redpanda broker’s node. The quotas are neither shared nor balanced between brokers.

Redpanda supports the following Kafka API-based quota types on clients:

Quota type Description

producer_byte_rate

Limit throughput of produce requests

consumer_byte_rate

Limit throughput of fetch requests

controller_mutation_rate

Limit rate of topic mutation requests, including create, add, and delete partition, in number of partitions per second

You can also apply a default quota for all other client requests that don’t have a specific quota based on an exact match or client_id prefix.

It is possible to create conflicting quotas if you configure the same quotas through both the Kafka API and a cluster configuration. Redpanda resolves these conflicts by following an order of preference in finding a matching quota for a request:

  1. Quota configured through the Kafka API for an exact match on client_id

  2. Quota configured through the Kafka API for a prefix match on client_id

  3. Quota configured through cluster configuration properties (kafka_client_group_byte_rate_quota, kafka_client_group_fetch_byte_rate_quota, deprecated starting in v24.2) for a prefix match on client_id

  4. Default quota configured through the Kafka API on client_id

  5. Default quota configured through cluster configuration properties (target_quota_byte_rate, target_fetch_quota_byte_rate, kafka_admin_topic_api_rate, deprecated starting in v24.2) on client_id

Redpanda recommends migrating over from cluster configuration-managed quotas to Kafka-compatible quotas. You can re-create the configuration-based quotas with rpk, and then remove the cluster configurations.

Individual client throughput limit

To view current throughput quotas set through the Kafka API, run rpk cluster quotas describe.

For example, to see the quotas for client ID consumer-1:

rpk cluster quotas describe --name client-id=consumer-1
bash
client-id=consumer-1
	producer_byte_rate=140000
bash

To set a throughput quota for a single client, use the rpk cluster quotas alter command.

rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id=consumer-1
bash
ENTITY                STATUS
client-id=consumer-1  OK
bash

Group of clients throughput limit

Alternatively, you can view or configure throughput quotas for a group of clients based on a match on client ID prefix. The following example sets the consumer_byte_rate quota to client IDs prefixed with consumer-:

rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id-prefix=consumer-
bash
A client group specified with client-id-prefix is not the equivalent of a Kafka consumer group. It is used only to match requests based on the client_id prefix. The client_id field is typically a configurable property when you create a client with Kafka libraries.

Default client throughput limit

You can apply default throughput limits to clients. Redpanda applies the default limits if no quotas are configured for a specific client_id or prefix.

To specify a produce quota of 1 GBps through the Kafka API (which means a 1 GBps limit across all produce requests sent to a single Redpanda broker), run:

rpk cluster quotas alter --default client-id --add producer_byte_rate=1000000000
bash

Bulk manage client throughput limits

To more easily manage multiple quotas, you can use the cluster quotas describe and cluster quotas import commands to do a bulk export and update.

For example, to export all client quotas in JSON format:

rpk cluster quotas describe --format json
bash

rpk cluster quotas import accepts the output string from rpk cluster quotas describe --format <json/yaml>:

rpk cluster quotas import --from '{"quotas":[{"entity":[{"name":"foo","type":"client-id"}],"values":[{"key":"consumer_byte_rate","values":"12123123"}]},{"entity":[{"name":"foo-","type":"client-id-prefix"}],"values":[{"key":"producer_byte_rate","values":"12123123"},{"key":"consumer_byte_rate","values":"4444444"}]}]}'
bash

You can also save the JSON or YAML output to a file and pass the file path in the --from flag.

Migrate cluster configuration quotas to Kafka API-based quotas

  1. Use rpk cluster config get to view current client quotas managed with cluster configuration. The following example shows how to retrieve the kafka_client_group_byte_rate_quota for two groups of producers:

    rpk cluster config get kafka_client_group_byte_rate_quota
    bash
    "kafka_client_group_byte_rate_quota": [
  {
    "group_name": "group_1",
    "clients_prefix": "producer_group_alone_producer",
    "quota": 10240
  },
  { "group_name": "group_2",
    "clients_prefix": "producer_group_multiple",
    "quota": 20480
  },
]
    bash

  2. Each client quota cluster property (deprecated in v24.2) corresponds to a quota type in Kafka. Check the corresponding rpk arguments to use when setting the new quota values:

    Cluster configuration property rpk cluster quotas arguments

    target_quota_byte_rate

    --default client-id --add producer_byte_rate=<producer-byte-rate-value>

    target_fetch_quota_byte_rate

    --default client-id --add consumer_byte_rate=<consumer-byte-rate-value>

    kafka_admin_topic_api_rate

    --default client-id --add controller_mutation_rate=<controller-mutation-rate-value>

    kafka_client_group_byte_rate_quota

    --name client-id-prefix=<prefix> --add producer_byte_rate=<group-producer-byte-rate-value>

    kafka_client_group_fetch_byte_rate_quota

    --name client-id-prefix=<prefix> --add consumer_byte_rate=<group-consumer-byte-rate-value>

    The client throughput quotas set through the Kafka API apply per broker, so you must convert the cluster configuration values that were applied on a per-shard (logical CPU core) basis. For example, if you set target_fetch_quota_byte_rate to 100 MBps/shard, and you run Redpanda on 16-core brokers, you can set the new consumer_byte_rate quota to 100 * 16 = 1600 MBps.

  3. Use rpk cluster quotas alter to set the corresponding client throughput quotas based on the Kafka API:

    rpk cluster quotas alter --name client-id-prefix=producer_group_alone_producer --add producer_byte_rate=<group-1-producer-byte-rate-value>
rpk cluster quotas alter --name client-id-prefix=producer_group_multiple --add producer_byte_rate=<group-2-producer-byte-rate-value>
    bash

    Replace the placeholder values with the new quota values, accounting for the conversion to per-broker limits. For example, 10240 * broker core count = new quota.

  4. Use rpk cluster config set to remove the configuration-based quotas:

    rpk cluster config set kafka_client_group_byte_rate_quota=
    bash

View throughput limits in Redpanda Console

You can also use Redpanda Console to view enforced limits. In the menu, go to Quotas.

Monitor client throughput

The following metrics are available on both the /public_metrics and /metrics endpoints to provide insights into client throughput quota usage:

The kafka_quotas logger provides details at the trace level on client quota throttling:

TRACE 2024-06-14 15:36:05,240 [shard  2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 1316, delay:184518451ns, capped_delay:184518451ns
TRACE 2024-06-14 15:36:05,240 [shard  2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:51256] throttle request:{snc:0, client:184}, enforce:{snc:-365123762, client:-365123762}, key:0, request_size:1316
TRACE 2024-06-14 15:37:44,835 [shard  2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 119, delay:0ns, capped_delay:0ns
TRACE 2024-06-14 15:37:59,195 [shard  2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 1316, delay:184518451ns, capped_delay:184518451ns
TRACE 2024-06-14 15:37:59,195 [shard  2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:58636] throttle request:{snc:0, client:184}, enforce:{snc:-14359, client:-14359}, key:0, request_size:1316
bash