Connect
Self-Managed Only

tigerbeetle_cdc

Introduced in version 4.65.0

Enables integration with TigerBeetle’s Change Data Capture (CDC) streaming for Redpanda Connect. This input component connects to a TigerBeetle cluster and continuously listens for real-time changes (such as account updates, transfers, and other ledger events). Each detected change is converted into a message and ingested into Redpanda Connect for further processing.

Use this input when you need to synchronize TigerBeetle’s financial ledger events with downstream systems, enable event-driven architectures, or build real-time analytics pipelines. Typical use cases include auditing, monitoring, fraud detection, and maintaining up-to-date replicas of financial data in other services.

  • Common

  • Advanced

input:
  label: ""
  tigerbeetle_cdc:
    cluster_id: "" # No default (required)
    addresses: [] # No default (required)
    progress_cache: "" # No default (required)
    rate_limit: ""
    event_count_max: 2730
    idle_interval_ms: 1000
    timestamp_initial: ""
    timeout_seconds: 15
    auto_replay_nacks: true
input:
  label: ""
  tigerbeetle_cdc:
    cluster_id: "" # No default (required)
    addresses: [] # No default (required)
    progress_cache: "" # No default (required)
    rate_limit: ""
    event_count_max: 2730
    idle_interval_ms: 1000
    timestamp_initial: ""
    timeout_seconds: 15
    auto_replay_nacks: true

Each message produced by this input is a JSON object representing a TigerBeetle CDC event. The structure includes details about the event type, ledger, transfer, debit and credit accounts, and associated timestamps. For example:

{
  "timestamp": "1745328372758695656",
  "type": "single_phase",
  "ledger": 2,
  "transfer": {
    "id": "9082709",
    "amount": "3794",
    "pending_id": "0",
    "user_data_128": "79248595801719937611592367840129079151",
    "user_data_64": "13615171707598273871",
    "user_data_32": 3229992513,
    "timeout": 0,
    "code": 20295,
    "flags": 0,
    "timestamp": "1745328372758695656"
  },
  "debit_account": {
    "id": "3750",
    "debits_pending": "0",
    "debits_posted": "8463768",
    "credits_pending": "0",
    "credits_posted": "8861179",
    "user_data_128": "118966247877720884212341541320399553321",
    "user_data_64": "526432537153007844",
    "user_data_32": 4157247332,
    "code": 1,
    "flags": 0,
    "timestamp": "1745328270103398016"
  },
  "credit_account": {
    "id": "6765",
    "debits_pending": "0",
    "debits_posted": "8669204",
    "credits_pending": "0",
    "credits_posted": "8637251",
    "user_data_128": "43670023860556310170878798978091998141",
    "user_data_64": "12485093662256535374",
    "user_data_32": 1924162092,
    "code": 1,
    "flags": 0,
    "timestamp": "1745328270103401031"
  }
}

Requirements

Before using the TigerBeetle CDC input, ensure that your environment meets the following requirements:

  • TigerBeetle cluster version 0.16.57 or later is required.

  • The TigerBeetle client version must not be newer than the cluster version to ensure compatibility.

  • Consumers must implement idempotency when processing messages to handle potential duplicates.

This component is available only in builds of Redpanda Connect that enable cgo. The standard release binaries, the Docker images, and the standard rpk connect plugin don’t include it. To use it, do one of the following:

  • Download the redpanda-connect-cgo_<version>_linux_amd64 archive from the GitHub releases page. This archive is available for Linux AMD64 only.

  • Build Redpanda Connect from source with CGO_ENABLED=1 go build -tags x_benthos_extra,timetzdata ./cmd/redpanda-connect. The build needs a C compiler.

The cgo archive and the source build both need the ZeroMQ library (libzmq), because they include the zmq4 components.

Metadata

This input adds the following metadata fields to each message:

  • event_type: One of "single_phase", "two_phase_pending", "two_phase_posted", "two_phase_voided", or "two_phase_expired".

  • ledger: The ledger code.

  • transfer_code: The transfer code.

  • debit_account_code: The debit account code.

  • credit_account_code: The credit account code.

  • timestamp: The unique event timestamp with nanosecond resolution.

  • timestamp_ms: The event timestamp with millisecond resolution.

Guarantees

This input guarantees at-least-once semantics, and makes a best effort to prevent duplicate messages. However, during crash recovery, it may replay unacknowledged messages that could have been already delivered to consumers.

It is the consumer’s responsibility to perform idempotency checks when processing messages.

Fields

addresses[]

A list of IP addresses of all the TigerBeetle replicas in the cluster. The order of addresses must correspond to the order of replicas.

Type: array<string>

auto_replay_nacks

Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.

Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.

Type: bool

Default: true

cluster_id

The TigerBeetle unique 128-bit cluster ID.

Type: string

event_count_max

The maximum number of events fetched from TigerBeetle per request. Must be greater than zero.

Type: int

Default: 2730

idle_interval_ms

The time interval in milliseconds to wait before querying again when the last request returned no events. Must be greater than zero.

Type: int

Default: 1000

progress_cache

A cache resource used to track progress by storing the last acknowledged timestamp. This allows Redpanda Connect to resume from the latest delivered event upon restart.

Type: string

rate_limit

An optional rate limit to throttle the number of requests made to TigerBeetle.

Type: string

Default: ""

timeout_seconds

The timeout in seconds, for querying the TigerBeetle cluster.

Requires version 4.82.0 or later.

Type: int

Default: 15

timestamp_initial

The initial timestamp to start extracting events from. If not defined, all events since the beginning will be included. Ignored if a more recent timestamp has already been acknowledged. This is a TigerBeetle timestamp with nanosecond precision.

Type: string

Default: ""