Configure Producers

The producer doesn’t wait for acknowledgments from the leader and doesn’t retry sending messages. This increases throughput and lowers latency of the system at the expense of durability and data loss.

This option allows a producer to immediately consider a message acknowledged when it is sent to the Redpanda broker. This means that a producer does not have to wait for any kind of response from the Redpanda broker. This is the least safe option because a leader-node crash can cause data loss if the data has not yet replicated to the other nodes in the replica set. However, this setting is useful in cases where you want to optimize for the highest throughput, and are willing to risk some data loss.

Because of the lack of guarantees, this setting is the most network bandwidth- efficient. This makes it useful for use cases like IoT/sensor data collection, where updates are periodic or stateless and you can afford some degree of data loss, but prefer gathering as much data as possible in a given time interval.

The producer waits for an acknowledgment from the leader, but it doesn’t wait for the leader to get acknowledgments from followers. This setting doesn’t prioritize throughput, latency, or durability. Instead, acks=1 attempts to provide a balance between all of them.

Replication is not guaranteed with this setting because it happens in the background, after the leader broker sends an acknowledgment to the producer. This setting could result in data loss if the leader broker crashes before any followers manage to replicate the message or if a majority of replica nodes go down at the same time before flushing the message to the disk.

The producer receives an acknowledgment after the majority of (implicitly, all) replicas acknowledge the message. This increases durability at the expense of lower throughput and increased latency.

Sometimes referred to as acks = -1, this option instructs the broker that replication is considered complete when the message has been replicated (and fsynced, meaning flushed to disk, making it durable in case of broker crashes) to the majority of the brokers responsible for the partition in the cluster. As soon as the fsync call is complete, the message is considered acknowledged and is made visible to readers.

This parameter controls the number of times a message is re-sent to the broker if the broker fails to acknowledge it. This is essentially the same as if the client application resends the erroneous message after receiving an error response. The default value of retries in most client libraries is 0. This means that if the send fails, the message is not re-sent at all.

If you increase this number to a higher value, be sure to take a look at the max.in.flight.requests.per.connection value as well, as leaving that parameter at its default value can potentially cause ordering issues in the target topic where the messages arrive. This occurs if two batches are sent to a single partition and the first fails and is retired, but the second succeeds so the records in the second batch may appear first.

This parameter controls how many unacknowledged messages are allowed to be sent to the broker simultaneously at any given time. The default value is

In cases when you configure the producer to be idempotent, up to five requests can be guaranteed to be in flight with the order preserved.

To enable idempotence, set enable.idempotence to true (the default) in your Redpanda configuration.

When idempotence is enabled, the producer will try to make sure that exactly one copy of every message is written to the broker. When set to false, the producer retries sending a message for any reason (such as transient errors like brokers not being available or not enough replicas exception), it can lead to duplicates.

In most client libraries enable.idempotence is set to true by default. Internally, this is implemented using a special identifier that is assigned to every producer (the producer ID or “PID”). This ID, along with a "sequence number", is included in every message that is being sent to the broker. The broker will check if the PID/sequence number combination is larger than the previous one and, if not, it will discard the message.

To guarantee true idempotent behavior, you must also set acks=all to ensure all brokers record messages in order, even in the event of node failures. In this configuration, both the producer and the broker prefer safety and durability over throughput.

Idempotence is only guaranteed within a session. A session starts once the producer is instantiated and a connection is established between the client and Redpanda broker. When the connection is closed, the session ends.

If your application code retries a request, the producer client will assign a new ID to that request, which may lead to duplicate messages.

buffer.memory is the value that controls the total amount of memory available to the producer for buffering. In a case where messages are sent faster than they can be delivered to the broker, the producer application may run out of memory, which will cause it to either block subsequent send calls or even throw an exception. The max.block.ms parameter controls the amount of time the producer will block before throwing an exception if it cannot immediately send messages to the broker.

batch.size controls the maximum size of coupled messages that can be batched together in one request. The producer will automatically put messages being sent to the same partition into one batch. This configuration parameter is given in bytes as opposed to the number of messages.

When the producer is gathering messages to assign to a batch, at some point it will hit this byte-size limit, which triggers it to send the batch to the broker. However, the producer does not necessarily wait (for as much time as set using linger.ms) until the batch is full. Sometimes, it can even send single-message batches. This means that setting the batch size too large is not necessarily undesirable, as it won’t cause throttling when sending messages; rather, it will only cause increased memory usage.

Conversely, setting the batch size too small can cause the producer to send batches of messages faster, which can cause network overhead, meaning a reduced throughput. The default value is usually 16384, but you can set this as low as 0, which turns off batching entirely.

linger.ms controls the maximum amount of time the producer will wait before sending out a batch of messages, if it is not already full. This means you can somewhat force the producer to make sure that batches are being filled up as efficiently as possible.

If you are willing to tolerate some latency, setting this value to a number larger than the default of 0 will cause the producer to send fewer, more efficient batches of messages. If you set the value to 0, there is still a high chance messages arrive around the same time to be batched together.

compression.type controls how the producer should compress a batch of messages before sending it to the broker. The default is none, which means the batch of messages will not be compressed at all. Compression occurs on full batches, so you can improve batching throughput by setting this parameter to use one of the available compression algorithms (along with increasing batch size). The available options are: zstd, lz4, gzip, and snappy.

Serializers are responsible for converting a message to a byte array. You can influence the speed/memory efficiency of your streaming setup by choosing one of the built-in serializers or writing a custom one. The performance consequences of using serializers is not typically significant.

For example, if you opt for the JSON serializer, you will have more data to transport with each message because every record will contain its schema in a verbose format, which impacts your compression speeds and network throughput. Alternatively, going with AVRO or Protobuf allows you to only define the schema in one place, while also enabling features like schema evolution.

Each time a broker receives a message with a skewed created timestamp that is outside a configured range, Redpanda increments the vectorized_kafka_rpc_produce_bad_create_time metric. Two cluster properties control this range. The minimum accepted value for both of these properties is five minutes. Any attempt to set a value lower than that is rejected by Redpanda. The properties are:

While not advised for typical use, Redpanda allows you to override the use of broker timestamps for retention policy. You accomplish this through use of the Admin API. Use the activate feature API to disable the broker_time_based_retention property.

If you disable this feature, make sure to specify your desired timestamp policy. This is stored in the log_message_timestamp_type cluster property. The timestamp policy defaults to CreateTime (client timestamp set by producer) but may be updated to LogAppendTime (server timestamp set by Redpanda).

From a producer perspective, when you want to get data into Redpanda as quickly as possible, you can maximize throughput in a variety of ways. You can set other components’ parameters, like experimenting with the topic partition size. You can also test acks settings.

For example, the quicker a producer receives a reply from the broker that the message has been committed, the sooner it can send the next message, which generally results in higher throughput. Hence, if you set acks=1, then the leader broker would not have to wait for replication to occur, and it can reply as soon as it is finished committing the message. As mentioned earlier, this can result in less durability overall.

Another option to explore is how the producer batches messages. Increasing the value of batch.size and linger.ms can increase throughput by making the producer add more messages into one batch before sending it to the broker and waiting until the batches can properly fill up. This approach negatively impacts latency though. By contrast, if you minimize linger.ms (for example, to 0) and batch.size to 1, you can achieve lower latency, but sacrifice throughput.

For applications where you must guarantee that there are no lost messages, duplicates, or service downtime, you can use higher durability acks settings. If you set acks=all, then the producer will wait for a majority of replicas to acknowledge the message before it can send the next message, resulting in lower latency, because there is more communication required between brokers. This approach can guarantee higher durability because the message will be replicated to all brokers.

You can also increase durability by increasing the number of retries the broker is allowed to make in case messages are not delivered successfully. The trade-off is that you may allow duplicates to enter the system and potentially alter the ordering of messages.