Idempotent producers

When a producer writes messages to a topic, each message should be recorded only once in the order in which it was sent. However, network issues such as a connection failure can result in a timeout, which prevents a write request from succeeding. In such cases, the client retries the write request until one of these events occurs:

  • The client receives an acknowledgment from the broker that the write was successful.

  • The retry limit is reached.

  • The message delivery timeout limit is reached.

Since there is no way to tell if the initial write request succeeded before the disruption, a retry can result in a duplicate message. A retry can also cause subsequent messages to be written out of order.

Idempotent producers prevent this problem by assigning a unique ID to every write request. The request ID consists of the producer ID and a sequence number. The sequence number identifies the order in which each write request was sent. If a retry results in a duplicate message, Redpanda detects and rejects the duplicate message and maintains the original order of the messages.

If new write requests continue while a previous request is being retried, the new requests are stored in the client’s memory in the order in which they were sent. The client must also retry these requests once the previous request is successful.

Enabling idempotence for producers

To make producers idempotent, the enable.idempotence property must be set to true in your producer configuration, as well as in the Redpanda cluster configuration, where it is set to true by default.

Some Kafka clients have enable.idempotence set to false by default. In this case, set the property to true by following the instructions for your particular client.

Idempotence is guaranteed within a session. A session starts once a producer is created and a connection is established between the client and the Kafka broker.

Idempotent producers retry unsuccessful write requests automatically. If you manually retry a write request, the client will assign a new ID to that request, which may lead to duplicate messages.

To disable idempotence (and risk duplicate messages as a result of retries), set enable_idempotence to false. For general instructions on how to edit any cluster property, refer to Configuring cluster properties.