Configure Client Connections

Optimize the availability of your clusters by configuring and tuning properties.

Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections.

  • Internal metrics that follow the vectorized_kafka_rpc_.*connect.* naming pattern provide details on Kafka client connection activity. For example, vectorized_kafka_rpc_active_connections reports the current number of active connections.

  • For Redpanda v25.3 and later, use rpk cluster connections list or the Admin API ListKafkaConnections endpoint to identify:

    • Which clients and applications are connected

    • Long-lived connections and long-running requests

    • Connections with no activity

    • Whether any clients are causing excessive load

    By reviewing connection details, you can make informed decisions about tuning connection limits and troubleshooting issues.

See also: ListKafkaConnections reference, Monitor Redpanda

Limit client connections

To mitigate the risk of a client creating too many connections and using too many system resources, you can configure a Redpanda cluster to impose limits on the number of client connections that can be created.

The following Redpanda cluster properties limit the number of connections:

  • kafka_connections_max_per_ip: Similar to Kafka’s max.connections.per.ip, this sets the maximum number of connections accepted per IP address by a broker.

  • kafka_connections_max_overrides: A list of IP addresses for which kafka_connections_max_per_ip is overridden and doesn’t apply.

  • kafka_connections_max: Similar to Kafka’s max.connections, this sets the maximum number of connections per broker.

Redpanda also provides properties to manage the rate of connection creation:

  • These connection limit properties are disabled by default. You must manually enable them.

  • The total number of connections is not equal to the number of clients, because a client can open multiple connections. As a conservative estimate, for a cluster with N brokers, plan for N + 2 connections per client.

Configure client reconnections

You can configure the Kafka client backoff and retry properties to change the default behavior of the clients to suit your failure requirements.

Set the following Kafka client properties on your application’s producer or consumer to manage client reconnections:

  • reconnect.backoff.ms: Amount of time to wait before attempting to reconnect to the broker. The default is 50 milliseconds.

  • reconnect.backoff.max.ms: Maximum amount of time in milliseconds to wait when reconnecting to a broker. The backoff increases exponentially for each consecutive connection failure, up to this maximum. The default is 1000 milliseconds (1 second).

Additionally, you can use Kafka properties to control message retry behavior. Delivery fails when either the delivery timeout or the number of retries is met.

  • delivery.timeout.ms: Amount of time for message delivery, so messages are not retried forever. The default is 120000 milliseconds (2 minutes).

  • retries: Number of times a producer can retry sending a message before marking it as failed. The default value is 2147483647 for Kafka >= 2.1, or 0 for Kafka <= 2.0.

  • retry.backoff.ms: Amount of time to wait before attempting to retry a failed request to a given topic partition. The default is 100 milliseconds.

See also