tigerbeetle_cdc
Introduced in version 4.65.0
Enables integration with TigerBeetle’s Change Data Capture (CDC) streaming for Redpanda Connect. This input component connects to a TigerBeetle cluster and continuously listens for real-time changes (such as account updates, transfers, and other ledger events). Each detected change is converted into a message and ingested into Redpanda Connect for further processing.
Use this input when you need to synchronize TigerBeetle’s financial ledger events with downstream systems, enable event-driven architectures, or build real-time analytics pipelines. Typical use cases include auditing, monitoring, fraud detection, and maintaining up-to-date replicas of financial data in other services.
# Config fields, showing default values
input:
label: ""
tigerbeetle_cdc:
cluster_id: "<your-cluster-id>" # Required: The unique 128-bit TigerBeetle cluster ID
addresses: # Required: List of IP addresses for all TigerBeetle replicas
- "<replica1-ip>"
- "<replica2-ip>"
- "<replica3-ip>"
progress_cache: "<your-cache-name>" # Required: Cache resource name to track progress
rate_limit: "" # Optional: Throttles request rate (defaults to no limit)
event_count_max: 2730 # Optional: Max events per request (default 2730)
idle_interval_ms: 1000 # Optional: Wait ms before querying again if no events (default 1000)
timestamp_initial: "" # Optional: TigerBeetle timestamp to start from (nanosecond precision)
auto_replay_nacks: true # Optional: Auto-replay failed batches (default true)Each message produced by this input is a JSON object representing a TigerBeetle CDC event. The structure includes details about the event type, ledger, transfer, debit and credit accounts, and associated timestamps. For example:
{
"timestamp": "1745328372758695656",
"type": "single_phase",
"ledger": 2,
"transfer": {
"id": "9082709",
"amount": "3794",
"pending_id": "0",
"user_data_128": "79248595801719937611592367840129079151",
"user_data_64": "13615171707598273871",
"user_data_32": 3229992513,
"timeout": 0,
"code": 20295,
"flags": 0,
"timestamp": "1745328372758695656"
},
"debit_account": {
"id": "3750",
"debits_pending": "0",
"debits_posted": "8463768",
"credits_pending": "0",
"credits_posted": "8861179",
"user_data_128": "118966247877720884212341541320399553321",
"user_data_64": "526432537153007844",
"user_data_32": 4157247332,
"code": 1,
"flags": 0,
"timestamp": "1745328270103398016"
},
"credit_account": {
"id": "6765",
"debits_pending": "0",
"debits_posted": "8669204",
"credits_pending": "0",
"credits_posted": "8637251",
"user_data_128": "43670023860556310170878798978091998141",
"user_data_64": "12485093662256535374",
"user_data_32": 1924162092,
"code": 1,
"flags": 0,
"timestamp": "1745328270103401031"
}
}Requirements
Before using the TigerBeetle CDC input, ensure that your environment meets the following requirements:
-
TigerBeetle cluster version 0.16.57 or later is required.
-
The TigerBeetle client version must not be newer than the cluster version to ensure compatibility.
-
This component is only available in cgo-enabled builds of Redpanda Connect (not in the Redpanda CLI or Docker image).
-
Consumers must implement idempotency when processing messages to handle potential duplicates.
Metadata
This input adds the following metadata fields to each message:
| Metadata Field | Description |
|---|---|
|
One of |
|
The ledger code. |
|
The transfer code. |
|
The debit account code. |
|
The credit account code. |
|
The unique event timestamp with nanosecond resolution. |
|
The event timestamp with millisecond resolution. |
Guarantees
This input guarantees at-least-once semantics, and makes a best effort to prevent duplicate messages. However, during crash recovery, it may replay unacknowledged messages that could have been already delivered to consumers.
It is the consumer’s responsibility to perform idempotency checks when processing messages.
Fields
addresses
A list of IP addresses of all the TigerBeetle replicas in the cluster. The order of addresses must correspond to the order of replicas.
Type: array
progress_cache
A cache resource used to track progress by storing the last acknowledged timestamp. This allows Redpanda Connect to resume from the latest delivered event upon restart.
Type: string
rate_limit
An optional rate limit to throttle the number of requests made to TigerBeetle.
Type: string
Default: ""
event_count_max
The maximum number of events fetched from TigerBeetle per request. Must be greater than zero.
Type: int
Default: 2730
idle_interval_ms
The time interval in milliseconds to wait before querying again when the last request returned no events. Must be greater than zero.
Type: int
Default: 1000
timestamp_initial
The initial timestamp to start extracting events from. If not defined, all CDC events available in the TigerBeetle cluster will be included.
Ignored if a more recent timestamp has already been acknowledged. This is a TigerBeetle timestamp with nanosecond precision.
Type: string
Default: ""
auto_replay_nacks
Whether messages that are rejected (nacked) at the output level are automatically replayed indefinitely. If the cause of rejection persists, this can result in backpressure. If set to false, these messages are deleted.
Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation.
Type: bool
Default: true
