Docs Self-Managed Develop Produce Data Idempotent Producers This is documentation for Self-Managed v24.1. To view the latest available version of the docs, see v24.2. Idempotent producers When a producer writes messages to a topic, each message should be recorded only once in the order in which it was sent. However, network issues such as a connection failure can result in a timeout, which prevents a write request from succeeding. In such cases, the client retries the write request until one of these events occurs: The client receives an acknowledgment from the broker that the write was successful. The retry limit is reached. The message delivery timeout limit is reached. Since there is no way to tell if the initial write request succeeded before the disruption, a retry can result in a duplicate message. A retry can also cause subsequent messages to be written out of order. Idempotent producers prevent this problem by assigning a unique ID to every write request. The request ID consists of the producer ID and a sequence number. The sequence number identifies the order in which each write request was sent. If a retry results in a duplicate message, Redpanda detects and rejects the duplicate message and maintains the original order of the messages. If new write requests continue while a previous request is being retried, the new requests are stored in the client’s memory in the order in which they were sent. The client must also retry these requests once the previous request is successful. Enable idempotence for producers To make producers idempotent, the enable.idempotence property must be set to true in your producer configuration, as well as in the Redpanda cluster configuration, where it is set to true by default. Some Kafka clients have enable.idempotence set to false by default. In this case, set the property to true by following the instructions for your particular client. Idempotence is guaranteed within a session. A session starts once a producer is created and a connection is established between the client and the Kafka broker. Idempotent producers retry unsuccessful write requests automatically. If you manually retry a write request, the client will assign a new ID to that request, which may lead to duplicate messages. To disable idempotence (and risk duplicate messages as a result of retries), set enable_idempotence to false. For instructions on how to edit any cluster property, see Configure cluster properties. Suggested labs Stream Stock Market Data from a CSV file Using Node.jsStream Stock Market Data from a CSV file Using PythonBuild a Chat Room Application with Redpanda and GolangBuild a Chat Room Application with Redpanda and JavaBuild a Chat Room Application with Redpanda and Node.jsBuild a Chat Room Application with Redpanda and PythonBuild a Chat Room Application with Redpanda and RustSee moreSearch all labs Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Configure Producers Consume Data