Manage Throughput
Redpanda supports applying throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. The purpose of this is to prevent unbounded network and disk usage of the broker by clients. Broker-wide limits apply to all clients connected to the broker and restrict total traffic on the broker. Client limits apply to a set of clients defined by their client_id
and help prevent a set of clients from starving other clients using the same broker.
Throughput throttling enforcement
As of v24.2, Redpanda enforces all throughput limits per broker, including client throughput. |
Throughput limits are enforced by applying backpressure to clients. When a connection is in breach of the throughput limit, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level. Redpanda starts by adding a throttle_time_ms
field to responses. If that isn’t honored, delays are inserted on the connection’s next read operation. The throttling delay may not exceed the limit set by max_kafka_throttle_delay_ms
.
Broker-wide throughput limits
Broker-wide throughput limits account for all Kafka API traffic going into or out of the broker, as data is produced to or consumed from a topic. The limit values represent the allowed rate of data in bytes per second passing through in each direction. Redpanda also provides administrators the ability to exclude clients from throughput throttling and to fine-tune which Kafka request types are subject to throttling limits.
Broker-wide throughput limit properties
The properties for broker-wide throughput quota balancing are configured at the cluster level, for all brokers in a cluster:
Cluster configuration property | Description |
---|---|
A broker’s total throughput limit for ingress Kafka traffic. |
|
A broker’s total throughput limit for egress Kafka traffic. |
|
List of clients for whom broker-wide limits do not apply |
|
List of Kafka request types subject to broker-wide throughput limits; defaults to |
|
The maximum delay inserted in the data path of Kafka API requests to throttle them down. Configuring this to be less than the Kafka client timeout can ensure that the delay that’s inserted won’t be long enough to cause a client timeout by itself. |
|
Client throughput limits
Redpanda provides configurable throughput quotas that apply to an individual client or a group of clients. You can apply a quota for an individual client based on an exact match with its client_id
, or a group of clients based on IDs that start with a given prefix.
As of v24.2, client throughput quotas are compatible with the AlterClientQuotas and DescribeClientQuotas Kafka APIs, and are separate from quotas configured through cluster configuration in earlier Redpanda versions. The client throughput quotas no longer apply on a per-shard basis, and now limit the rates across a Redpanda broker’s node. The quotas are neither shared nor balanced between brokers.
Redpanda supports the following Kafka API-based quota types on clients:
Quota type | Description |
---|---|
|
Limit throughput of produce requests |
|
Limit throughput of fetch requests |
|
Limit rate of topic mutation requests, including create, add, and delete partition, in number of partitions per second |
You can also apply a default quota for all other client requests that don’t have a specific quota based on an exact match or client_id
prefix.
It is possible to create conflicting quotas if you configure the same quotas through both the Kafka API and a cluster configuration. Redpanda resolves these conflicts by following an order of preference in finding a matching quota for a request:
-
Quota configured through the Kafka API for an exact match on
client_id
-
Quota configured through the Kafka API for a prefix match on
client_id
-
Quota configured through cluster configuration properties (
kafka_client_group_byte_rate_quota
,kafka_client_group_fetch_byte_rate_quota
, deprecated starting in v24.2) for a prefix match onclient_id
-
Default quota configured through the Kafka API on
client_id
-
Default quota configured through cluster configuration properties (
target_quota_byte_rate
,target_fetch_quota_byte_rate
,kafka_admin_topic_api_rate
, deprecated starting in v24.2) onclient_id
Redpanda recommends migrating over from cluster configuration-managed quotas to Kafka-compatible quotas. You can re-create the configuration-based quotas with rpk
, and then remove the cluster configurations.
Individual client throughput limit
To view current throughput quotas set through the Kafka API, run rpk cluster quotas describe
.
For example, to see the quotas for client ID consumer-1
:
rpk cluster quotas describe --name client-id=consumer-1
client-id=consumer-1
producer_byte_rate=140000
To set a throughput quota for a single client, use the rpk cluster quotas alter
command.
rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id=consumer-1
ENTITY STATUS
client-id=consumer-1 OK
Group of clients throughput limit
Alternatively, you can view or configure throughput quotas for a group of clients based on a match on client ID prefix. The following example sets the consumer_byte_rate
quota to client IDs prefixed with consumer-
:
rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id-prefix=consumer-
A client group specified with client-id-prefix is not the equivalent of a Kafka consumer group. It is used only to match requests based on the client_id prefix. The client_id field is typically a configurable property when you create a client with Kafka libraries.
|
Default client throughput limit
You can apply default throughput limits to clients. Redpanda applies the default limits if no quotas are configured for a specific client_id
or prefix.
To specify a produce quota of 1 GBps through the Kafka API (which means a 1 GBps limit across all produce requests sent to a single Redpanda broker), run:
rpk cluster quotas alter --default client-id --add producer_byte_rate=1000000000
Bulk manage client throughput limits
To more easily manage multiple quotas, you can use the cluster quotas describe
and cluster quotas import
commands to do a bulk export and update.
For example, to export all client quotas in JSON format:
rpk cluster quotas describe --format json
rpk cluster quotas import
accepts the output string from rpk cluster quotas describe --format <json/yaml>
:
rpk cluster quotas import --from '{"quotas":[{"entity":[{"name":"foo","type":"client-id"}],"values":[{"key":"consumer_byte_rate","values":"12123123"}]},{"entity":[{"name":"foo-","type":"client-id-prefix"}],"values":[{"key":"producer_byte_rate","values":"12123123"},{"key":"consumer_byte_rate","values":"4444444"}]}]}'
You can also save the JSON or YAML output to a file and pass the file path in the --from
flag.
Migrate cluster configuration quotas to Kafka API-based quotas
-
Use
rpk cluster config get
to view current client quotas managed with cluster configuration. The following example shows how to retrieve thekafka_client_group_byte_rate_quota
for two groups of producers:rpk cluster config get kafka_client_group_byte_rate_quota
bash"kafka_client_group_byte_rate_quota": [ { "group_name": "group_1", "clients_prefix": "producer_group_alone_producer", "quota": 10240 }, { "group_name": "group_2", "clients_prefix": "producer_group_multiple", "quota": 20480 }, ]
bash -
Each client quota cluster property (deprecated in v24.2) corresponds to a quota type in Kafka. Check the corresponding
rpk
arguments to use when setting the new quota values:Cluster configuration property rpk cluster quotas
argumentstarget_quota_byte_rate
--default client-id --add producer_byte_rate=<producer-byte-rate-value>
target_fetch_quota_byte_rate
--default client-id --add consumer_byte_rate=<consumer-byte-rate-value>
kafka_admin_topic_api_rate
--default client-id --add controller_mutation_rate=<controller-mutation-rate-value>
kafka_client_group_byte_rate_quota
--name client-id-prefix=<prefix> --add producer_byte_rate=<group-producer-byte-rate-value>
kafka_client_group_fetch_byte_rate_quota
--name client-id-prefix=<prefix> --add consumer_byte_rate=<group-consumer-byte-rate-value>
The client throughput quotas set through the Kafka API apply per broker, so you must convert the cluster configuration values that were applied on a per-shard (logical CPU core) basis. For example, if you set
target_fetch_quota_byte_rate
to 100 MBps/shard, and you run Redpanda on 16-core brokers, you can set the new consumer_byte_rate quota to 100 * 16 = 1600 MBps. -
Use
rpk cluster quotas alter
to set the corresponding client throughput quotas based on the Kafka API:rpk cluster quotas alter --name client-id-prefix=producer_group_alone_producer --add producer_byte_rate=<group-1-producer-byte-rate-value> rpk cluster quotas alter --name client-id-prefix=producer_group_multiple --add producer_byte_rate=<group-2-producer-byte-rate-value>
bashReplace the placeholder values with the new quota values, accounting for the conversion to per-broker limits. For example, 10240 * broker core count = new quota.
-
Use
rpk cluster config set
to remove the configuration-based quotas:rpk cluster config set kafka_client_group_byte_rate_quota=
bash
View throughput limits in Redpanda Console
You can also use Redpanda Console to view enforced limits. In the menu, go to Quotas.
Monitor client throughput
The following metrics are available on both the /public_metrics
and /metrics
endpoints to provide insights into client throughput quota usage:
-
Client quota throughput per rule and quota type:
-
/public_metrics
-redpanda_kafka_quotas_client_quota_throughput
-
-
Client quota throttling delay per rule and quota type, in seconds:
-
/public_metrics
-redpanda_kafka_quotas_client_quota_throttle_time
-
/metrics
-vectorized_kafka_quotas_client_quota_throttle_time
-
The kafka_quotas
logger provides details at the trace level on client quota throttling:
TRACE 2024-06-14 15:36:05,240 [shard 2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 1316, delay:184518451ns, capped_delay:184518451ns
TRACE 2024-06-14 15:36:05,240 [shard 2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:51256] throttle request:{snc:0, client:184}, enforce:{snc:-365123762, client:-365123762}, key:0, request_size:1316
TRACE 2024-06-14 15:37:44,835 [shard 2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 119, delay:0ns, capped_delay:0ns
TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 1316, delay:184518451ns, capped_delay:184518451ns
TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:58636] throttle request:{snc:0, client:184}, enforce:{snc:-14359, client:-14359}, key:0, request_size:1316