Consumer Offsets

Redpanda supports __consumer_offsets, which is a private topic on a Redpanda node. The __consumer_offsets topic stores committed offsets from each Kafka consumer that is attached to Redpanda.

Redpanda exposes the __consumer_offsets key to enable the many tools in the Kafka ecosystem that rely on this value for their operation, providing greater ecosystem interoperability with environments and applications.

Kafka consumer tracks the maximum offset it has consumed in each partition and has the capability to commit offsets so that it can resume from those offsets in the event of a restart. Kafka provides the option to store all the offsets for a given consumer group in a designated broker (for that group) called the "group coordinator". In other words, any consumer instance in that consumer group should send its offset commits and fetches to that group coordinator (broker).

A consumer group is a set of consumers that cooperate to consume data from some topics. The partitions of all the topics are divided among the consumers in the group. As new group members arrive and old members leave, the partitions are re-assigned so that each member receives a proportional share of the partitions.

When the group coordinator receives an OffsetCommitRequest, it appends the request to a special compacted Kafka topic named __consumer_offsets. The broker sends a successful offset commit response to the consumer only after all the replicas of the offsets topic receive the offsets. In case the offsets fail to replicate within a configurable timeout, the offset commit will fail and the consumer may retry the commit after backing off. The brokers periodically compact the offsets topic because it only needs to maintain the most recent offset commit per partition. The coordinator also caches the offsets in an in-memory table to serve offset fetches quickly.

When stored in a partition, a sequential number is assigned to each event for order. This number is the offset. The offset allows consumer event processing to start or continue from where it left off. So, if you have a consumer application outage that prevents your application from receiving events, you can use the consumer offset to select and send only the events that occurred when the application was down. By default, the offset starts at 0, but you can manually specify a specific start value.