Collapse navigation tree

Collapse

About Client Throughput Quotas

Redpanda uses throughput quotas to limit the rate of produce and consume requests from clients. Understanding how quotas work helps you prevent individual clients from disproportionately consuming resources and causing performance degradation for other clients (also known as the "noisy-neighbor" problem), and ensure fair resource sharing across users and applications.

After reading this page, you will be able to:

  • Describe the difference between user-based and client ID-based quotas

  • Determine which quota type to use for your use case

  • Explain quota precedence rules and how Redpanda tracks quota usage

To configure and manage throughput quotas, see Manage Throughput.

Throughput control overview

Redpanda provides two ways to control throughput:

  • Broker-wide limits: Configured using cluster properties. For details, see Broker-wide throughput limits.

  • Client throughput quotas: Configured using the Kafka API. Client quotas enable per-user and per-client rate limiting with fine-grained control through entity hierarchy and precedence rules. This page focuses on client quotas.

Supported quota types

Redpanda supports three Kafka API-based quota types:

Quota type Description

producer_byte_rate

Limit throughput of produce requests (bytes per second)

consumer_byte_rate

Limit throughput of fetch requests (bytes per second)

controller_mutation_rate

Limit rate of topic mutation requests (partitions created or deleted per second)

All quota types can be applied to groups of client connections based on user principals, client IDs, or combinations of both.

Quota entities

Redpanda uses two pieces of identifying information from each client connection to determine which quota applies:

  • Client ID: An ID that clients self-declare. Quotas can target an exact client ID (client-id) or a prefix (client-id-prefix). Multiple client connections that share a client ID or ID prefix are grouped into a single quota entity.

  • User principal: An authenticated identity verified through SASL, mTLS, or OIDC. Connections that share the same user are considered one entity.

You can configure quotas that target either entity type, or combine both for fine-grained control.

Client ID-based quotas

Client ID-based quotas apply to clients identified by their client-id field, which is set by the client application. The client ID is typically a configurable property when you create a client with Kafka libraries. When using client ID-based quotas, multiple clients using the same client ID share the same quota tracking.

Client ID-based quotas rely on clients honestly reporting their identity and correctly setting the client-id property. This makes client ID-based quotas unsuitable for guaranteeing isolation between tenants.

Use client ID-based quotas when:

  • Authentication is not enabled.

  • Grouping by application or service name is sufficient.

  • You operate a single-tenant environment where all clients are trusted.

  • You need simple rate limiting without user-level isolation.

User-based quotas

User-based quotas require authentication to be enabled on your cluster.

User-based quotas apply to authenticated user principals. Each user has a separate quota, providing a way to limit the impact of individual users on the cluster.

User-based quotas rely on Redpanda’s authentication system to verify user identity. The user principal is extracted from SASL credentials, mTLS certificates, or OIDC tokens and cannot be forged by clients.

Use user-based quotas when:

  • You operate a multi-tenant environment, such as SaaS platforms or enterprises with departments.

  • You require isolation between users or tenants, to avoid noisy-neighbor issues.

  • You need per-user billing or metering.

Combined user and client quotas

You can combine user and client identities for fine-grained control over specific (user, client) combinations.

Use combined quotas when:

  • You need fine-grained control, for example: user alice using a specific application.

  • Different rate limits apply to different apps used by the same user. For example, alice's payment-processor gets 10 MB/s, but alice's analytics-consumer gets 50 MB/s. See Quota precedence and tracking for examples.

Quota precedence and tracking

When a request arrives, Redpanda resolves which quota to apply by matching the request’s authenticated user principal and client ID against configured quotas. Redpanda applies the most specific match, using the precedence order in the following table (highest priority first).

The precedence level that matches also determines how quota usage is tracked. Redpanda tracks quota usage using a tracker key that determines which connections share the same quota bucket. How connections are grouped into buckets depends on the type of entity the quota targets.

To get independent quota tracking per user and client ID combination, configure quotas that include both dimensions, such as /config/users/<user>/clients/<client-id> or /config/users/<default>/clients/<default>.

Table 1. Quota precedence, tracking, and isolation by configuration level
Level Match type Config path Tracker key Isolation behavior

1

Exact user + exact client

/config/users/<user>/clients/<client-id>

(user, client-id)

Each unique (user, client-id) pair tracked independently

2

Exact user + client prefix

/config/users/<user>/client-id-prefix/<prefix>

(user, client-id-prefix)

Clients matching the prefix share tracking within that user

3

Exact user + default client

/config/users/<user>/clients/<default>

(user, client-id)

Each unique (user, client-id) pair tracked independently

4

Exact user only

/config/users/<user>

user

All clients for that user share a single tracking bucket

5

Default user + exact client

/config/users/<default>/clients/<client-id>

(user, client-id)

Each unique (user, client-id) pair tracked independently

6

Default user + client prefix

/config/users/<default>/client-id-prefix/<prefix>

(user, client-id-prefix)

Clients matching the prefix share tracking within each user

7

Default user + default client

/config/users/<default>/clients/<default>

(user, client-id)

Each unique (user, client-id) pair tracked independently

8

Default user only

/config/users/<default>

user

All clients for each user share a single tracking bucket (per user)

9

Exact client only

/config/clients/<client-id>

client-id

All users with that client ID share a single tracking bucket

10

Client prefix only

/config/client-id-prefix/<prefix>

client-id-prefix

All clients matching the prefix share a single bucket across all users

11

Default client only

/config/clients/<default>

client-id

Each unique client ID tracked independently

12

No quota configured

N/A

N/A

No tracking / unlimited throughput
The <default> entity matches any user or client that doesn’t have a more specific quota configured. This is different from an empty/unauthenticated user (user=""), or undeclared client ID (client-id=""), which are treated as specific entities.

Unauthenticated connections

Unauthenticated connections have an empty user principal (user="") and are not treated as user=<default>.

Unauthenticated connections:

  • Fall back to client-only quotas.

  • Have unlimited throughput only if no client-only quota matches.

Example: Precedence resolution

Given these configured quotas:

rpk cluster quotas alter --add consumer_byte_rate=5000000 --name user=alice --name client-id=app-1
rpk cluster quotas alter --add consumer_byte_rate=10000000 --name user=alice
rpk cluster quotas alter --add consumer_byte_rate=20000000 --name client-id=app-1
User + Client ID Precedence match

user=alice, client-id=app-1

Level 1: Exact user + exact client

user=alice, client-id=app-2

Level 4: Exact user only

user=bob, client-id=app-1

Level 9: Exact client only

user=bob, client-id=app-2

Level 12: No quota configured

When no quota matches (level 12), the connection is not throttled.

Example: User-only quota

If you configure a 10 MB/s produce quota for user alice:

rpk cluster quotas alter --add producer_byte_rate=10000000 --name user=alice

Then alice connecting with client ID app-1 and alice connecting with client ID app-2 share the same 10 MB/s produce limit.

To give each of alice's clients an independent 10 MB/s limit, configure:

rpk cluster quotas alter --add producer_byte_rate=10000000 --name user=alice --default client-id

Example: User default quota

If you configure a default 10 MB/s produce quota for all users:

rpk cluster quotas alter --add producer_byte_rate=10000000 --default user

This quota applies to all users who don’t have a more specific quota configured. Each user is tracked independently: alice gets her own 10 MB/s bucket, bob gets his own 10 MB/s bucket, and so on.

Within each user, all client ID values share that user’s bucket. alice connecting with client ID app-1 and alice connecting with client ID app-2 share the same 10 MB/s produce limit, while bob's connections have a separate 10 MB/s limit.

Throughput throttling enforcement

As of v24.2, Redpanda enforces all throughput limits per broker, including client throughput.

Redpanda enforces throughput limits by applying backpressure to clients. When a connection exceeds its throughput limit, Redpanda throttles the connection to bring the rate back within the allowed level:

  1. Redpanda adds a throttle_time_ms field to responses, indicating how long the client should wait.

  2. If the client doesn’t honor the throttle time, Redpanda inserts delays on the connection’s next read operation.

The throttling delay may not exceed the limit set by the max_kafka_throttle_delay_ms tunable property.

Default behavior

Quotas are opt-in restrictions and not enforced by default. When no quotas are configured, clients have unlimited throughput.

Next steps