Idempotent Producers
When a producer writes messages to a topic, each message should be recorded only once in the order in which it was sent. However, a disruption such as a timeout or a transient connection failure can prevent a write request from succeeding. In such cases, the client retries the write request until one of these events occurs:
-
The client receives an acknowledgment from the broker that the write was successful.
-
The retry limit is reached.
-
The message delivery timeout limit is reached.
Since there is no way to tell if the initial write request succeeded before the disruption, a retry can result in a duplicate message. A retry can also cause subsequent messages to be written out of order.
Idempotent producers prevent this problem by assigning a unique ID to every write request. The request ID consists of the producer ID and a sequence number. The sequence number identifies the order in which each write request was sent. If a retry results in a duplicate message, Redpanda detects and rejects the duplicate message and maintains the original order of the messages.
If new write requests continue while a previous request is being retried, the new requests are stored in the client’s memory in the order in which they were received. The client must also retry these requests once the previous request is successful.
Enabling idempotence for producers
The enable_idempotence property is enabled by default. This property is set at the cluster level, which means the setting applies to all nodes in the cluster.
Certain client-side settings are required in order for idempotence to work properly. These settings are configured automatically when idempotence is enabled.
Idempotence is guaranteed within a session. A session starts once a producer is created and a connection is established between the client and the Kafka broker.
| Idempotent producers retry unsuccessful write requests automatically. If you manually retry a write request, the client will assign a new ID to that request, which may lead to duplication. |
To disable idempotence (and risk duplicate messages as a result of retries), set enable_idempotence to false. For general instructions on how to edit any cluster property, refer to Configuring cluster properties.