broker
A meta-output that routes messages to child outputs using a range of brokering patterns. Unlike regular outputs, broker doesn’t send messages anywhere by itself. Instead, it wraps other outputs and controls how messages are delivered across them.
Use broker to fan out the same message to multiple destinations (for example, publishing events to Kafka while also writing them to a database), or to distribute messages across a pool of outputs for load balancing or throughput scaling. The delivery pattern determines whether each message is written to all outputs or routed to a single output, and whether writes happen in parallel or in sequence.
The name broker refers to the brokering delivery pattern, not a Redpanda broker (cluster node).
|
-
Common
-
Advanced
output:
label: ""
broker:
pattern: fan_out
outputs: [] # No default (required)
batching:
count: 0
byte_size: 0
period: ""
check: ""output:
label: ""
broker:
copies: 1
pattern: fan_out
outputs: [] # No default (required)
batching:
count: 0
byte_size: 0
period: ""
check: ""
processors: [] # No default (optional)Processors can be listed to apply across individual outputs or all outputs:
output:
broker:
pattern: fan_out
outputs:
- resource: foo
- resource: bar
# Processors only applied to messages sent to bar.
processors:
- resource: bar_processor
# Processors applied to messages sent to all brokered outputs.
processors:
- resource: general_processorFields
batching
Configure a batching policy.
Type: object
# Examples:
batching:
byte_size: 5000
count: 0
period: 1s
# ---
batching:
count: 10
period: 1s
# ---
batching:
check: this.contains("END BATCH")
count: 0
period: 1mbatching.byte_size
The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).
Set to 0 to disable size-based batching. When disabled, messages are flushed based on other conditions (such as count or period).
Type: int
Default: 0
batching.check
A Bloblang query that returns a boolean value indicating whether a message should end a batch.
Type: string
Default: ""
# Examples:
check: this.type == "end_of_transaction"batching.count
The number of messages at which the batch is flushed. Set to 0 to disable count-based batching.
Type: int
Default: 0
batching.period
The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s. Supported time units are ns, us, ms, s, m, and h.
Type: string
Default: ""
# Examples:
period: 1s
# ---
period: 1m
# ---
period: 500msbatching.processors[]
A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.
Type: array<processor>
# Examples:
processors:
- archive:
format: concatenate
# ---
processors:
- archive:
format: lines
# ---
processors:
- archive:
format: json_arrayPatterns
The broker pattern determines how messages are distributed across outputs. Use fan_out (the default) when every output should receive every message. Use round_robin or greedy when you want to distribute messages across outputs for load balancing rather than duplication.
The available patterns are:
fan_out
With the fan out pattern all outputs will be sent every message that passes through Redpanda Connect in parallel.
If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates.
Sometimes it is useful to disable the back pressure or retries of certain fan out outputs and instead drop messages that have failed or were blocked. In this case you can wrap outputs with a drop_on output.
fan_out_fail_fast
The same as the fan_out pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs.
fan_out_sequential
Similar to the fan out pattern except outputs are written to sequentially, meaning an output is only written to once the preceding output has confirmed receipt of the same message.
If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates.
fan_out_sequential_fail_fast
The same as the fan_out_sequential pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs.
round_robin
With the round robin pattern each message will be assigned a single output following their order. If an output applies back pressure it will block all subsequent messages. If an output fails to send a message then the message will be re-attempted with the next input, and so on.
greedy
The greedy pattern results in higher output throughput at the cost of potentially disproportionate message allocations to those outputs. Each message is sent to a single output, which is determined by allowing outputs to claim messages as soon as they are able to process them. This results in certain faster outputs potentially processing more messages at the cost of slower outputs.
