Message Delivery Semantics

An ack() function is associated with every batch of messages. This ack() function is a callback function provided by the input. It’s the output’s job to call the ack() function to inform the input of the success ("ack") or failure (negative ack or "nack") of the batch. The input can then decide what to do, for example, auto-retry nacks, propagate the nack upstream to input connectors that support it (Google PubSub), or drop the batch. When connecting to at-least-once inputs and outputs, Redpanda Connect guarantees at-least-once delivery without having to persist messages during transit.

Key benefits:

At-least-once delivery in a Redpanda Connect pipeline means every message is delivered at-least-once even if failures occur (which could result in some duplicates, but no lost data). Introducing a distributed or externally persisted state into a pipeline can compromise this guarantee. The pipeline’s built-in fault tolerance is based on the assumption that if an output fails, the message will be retried or reprocessed upon recovery. However, an external state (such as a remote cache or database used within the pipeline) may not reset on failure. It could retain an indication that a message was processed even if that message was never successfully processed by the output, and this mismatch can lead to message loss during recovery.

For example, a pipeline uses a distributed cache to keep track of message IDs that have been seen in order to filter out duplicates. When a new message arrives, its ID is added to the cache immediately to mark it as processed. Now imagine that the pipeline’s output connector fails or the pipeline instance crashes just after updating the cache, but before the message actually reaches the destination. When the system comes back up, it will retry processing any unacknowledged messages, but because the cache still contains that message ID from the earlier attempt, the pipeline treats the message as a duplicate and drops it. As a result, that message is never delivered by the output at all, which is a direct violation of at-least-once guarantees.

One way to mitigate this risk is to avoid using a state that outlives the pipeline process. For instance, keeping the cache in-memory (local to each pipeline instance) means it won’t preserve entries through a crash. If the process restarts, an in-memory cache would start fresh, allowing the system to reprocess any message that didn’t complete earlier instead of mistakenly skipping it. If you combine this with consistent partitioning of messages by key across horizontally scaled pipeline instances (so that each key is always handled by the same instance), you can still perform de-duplication within each partition while keeping state isolated per instance. In this setup, a failure that restarts one instance will clear that instance’s cache, and upon recovery it will reprocess its pending messages without interference from stale state.

Even with such mitigations, if reliable delivery is paramount, a safer approach is to avoid relying on any external persistent state for eliminating duplicates within the pipeline. Instead, let the pipeline run with its native at-least-once behavior (allowing potential duplicates on retries) and design the downstream systems to handle duplicates idempotently.

Redpanda Connect embraces a declarative and opt-in error-handling strategy:

This model provides flexibility in building resilient pipelines without introducing unnecessary complexity. Refer to the specific Error Handling documentation for further details and examples.