Connect
Self-Managed Only

tigerbeetle_cdc

Streams change data capture (CDC) events from a TigerBeetle cluster.

Introduced in version 4.65.0.

Listens to a TigerBeetle cluster and creates a message for each change. Each change is a transfer event, such as a single-phase transfer or a step of a two-phase transfer, together with the state of its debit and credit accounts after the change.

Each message is a JSON object that includes the event timestamp and type, the ledger, the transfer, and the debit and credit accounts, like:

{
  "timestamp": "1745328372758695656",
  "type": "single_phase",
  "ledger": 2,
  "transfer": {
    "id": "9082709",
    "amount": "3794",
    "pending_id": "0",
    "user_data_128": "79248595801719937611592367840129079151",
    "user_data_64": "13615171707598273871",
    "user_data_32": 3229992513,
    "timeout": 0,
    "code": 20295,
    "flags": 0,
    "timestamp": "1745328372758695656"
  },
  "debit_account": {
    "id": "3750",
    "debits_pending": "0",
    "debits_posted": "8463768",
    "credits_pending": "0",
    "credits_posted": "8861179",
    "user_data_128": "118966247877720884212341541320399553321",
    "user_data_64": "526432537153007844",
    "user_data_32": 4157247332,
    "code": 1,
    "flags": 0,
    "timestamp": "1745328270103398016"
  },
  "credit_account": {
    "id": "6765",
    "debits_pending": "0",
    "debits_posted": "8669204",
    "credits_pending": "0",
    "credits_posted": "8637251",
    "user_data_128": "43670023860556310170878798978091998141",
    "user_data_64": "12485093662256535374",
    "user_data_32": 1924162092,
    "code": 1,
    "flags": 0,
    "timestamp": "1745328270103401031"
  }
}

For more information, see the TigerBeetle CDC documentation.

Metadata

This input adds the following metadata fields to each message:

  • event_type: One of "single_phase", "two_phase_pending", "two_phase_posted", "two_phase_voided", or "two_phase_expired".

  • ledger: The ledger code.

  • transfer_code: The transfer code.

  • debit_account_code: The debit account code.

  • credit_account_code: The credit account code.

  • timestamp: The unique event timestamp with nanosecond resolution.

  • timestamp_ms: The event timestamp with millisecond resolution.

Guarantees

This input guarantees at-least-once semantics, and makes a best effort to prevent duplicate messages. However, during crash recovery, it may replay unacknowledged messages that could have been already delivered to consumers.

It is the consumer’s responsibility to perform idempotency checks when processing messages.

Upgrading

The TigerBeetle client version must not be newer than the cluster version, as it will fail with an error message if so.

Requires TigerBeetle cluster version 0.16.57 or greater.

  • Common

  • Advanced

input:
  label: ""
  tigerbeetle_cdc:
    cluster_id: "" # No default (required)
    addresses: [] # No default (required)
    progress_cache: "" # No default (required)
    rate_limit: ""
    event_count_max: 2730
    idle_interval_ms: 1000
    timestamp_initial: ""
    timeout_seconds: 15
    auto_replay_nacks: true
input:
  label: ""
  tigerbeetle_cdc:
    cluster_id: "" # No default (required)
    addresses: [] # No default (required)
    progress_cache: "" # No default (required)
    rate_limit: ""
    event_count_max: 2730
    idle_interval_ms: 1000
    timestamp_initial: ""
    timeout_seconds: 15
    auto_replay_nacks: true

This component is available only in builds of Redpanda Connect that enable cgo. The standard release binaries, the Docker images, and the standard rpk connect plugin don’t include it. To use it, do one of the following:

  • Download the redpanda-connect-cgo_<version>_linux_amd64 archive from the GitHub releases page. This archive is available for Linux AMD64 only.

  • Build Redpanda Connect from source with CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata ./cmd/redpanda-connect. The build needs a C compiler.

The cgo archive and the source build both need the ZeroMQ library (libzmq), because they include the zmq4 components.

Fields

addresses[]

A list of IP addresses of all the TigerBeetle replicas in the cluster. The order of addresses must correspond to the order of replicas.

Type: array<string>

auto_replay_nacks

Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.

Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.

Type: bool

Default: true

cluster_id

The TigerBeetle unique 128-bit cluster ID.

Type: string

event_count_max

The maximum number of events fetched from TigerBeetle per request. Must be greater than zero.

Type: int

Default: 2730

idle_interval_ms

The time interval in milliseconds to wait before querying again when the last request returned no events. Must be greater than zero.

Type: int

Default: 1000

progress_cache

A cache resource used to track progress by storing the last acknowledged timestamp. This allows Redpanda Connect to resume from the latest delivered event upon restart.

Type: string

rate_limit

An optional rate limit to throttle the number of requests made to TigerBeetle.

Type: string

Default: ""

timeout_seconds

The timeout in seconds, for querying the TigerBeetle cluster.

Requires version 4.82.0 or later.

Type: int

Default: 15

timestamp_initial

The initial timestamp to start extracting events from. If not defined, all events since the beginning will be included. Ignored if a more recent timestamp has already been acknowledged. This is a TigerBeetle timestamp with nanosecond precision.

Type: string

Default: ""