Cloud
Also in Self-Managed

kafka_franz

Deprecated in 4.68.0

This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified redpanda input and redpanda output components.

The kafka_franz output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgments back to the input. This output often outperforms the traditional kafka output, as well as providing more useful logs and error messages.

This output uses the Franz Kafka client library.

  • Common

  • Advanced

output:
  label: ""
  kafka_franz:
    seed_brokers: [] # No default (required)
    topic: "" # No default (required)
    key: "" # No default (optional)
    partition: "" # No default (optional)
    metadata:
      include_prefixes: []
      include_patterns: []
    max_in_flight: 10
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
output:
  label: ""
  kafka_franz:
    seed_brokers: [] # No default (required)
    client_id: redpanda-connect
    tls:
      enabled: false
      skip_cert_verify: false
      enable_renegotiation: false
      root_cas: ""
      root_cas_file: ""
      client_certs: []
    sasl: [] # No default (optional)
    metadata_max_age: 1m
    request_timeout_overhead: 10s
    conn_idle_timeout: 20s
    tcp:
      connect_timeout: 0s
      keep_alive:
        idle: 15s
        interval: 15s
        count: 9
      tcp_user_timeout: 0s
    topic: "" # No default (required)
    key: "" # No default (optional)
    partition: "" # No default (optional)
    metadata:
      include_prefixes: []
      include_patterns: []
    timestamp_ms: "" # No default (optional)
    max_in_flight: 10
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
      processors: [] # No default (optional)
    partitioner: "" # No default (optional)
    idempotent_write: true
    acks: all
    compression: "" # No default (optional)
    allow_auto_topic_creation: true
    timeout: 10s
    max_message_bytes: 1MiB
    broker_write_max_bytes: 100MiB
    max_buffered_records: 10000
    max_buffered_bytes: "0"
    max_in_flight_requests: 1
    record_retries: 0
    record_delivery_timeout: 0s

Fields

acks

The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When idempotent_write is enabled this must be set to all.

Type: string

Default: all

Option Summary

all

Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled.

leader

Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication.

none

Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost.

allow_auto_topic_creation

Enables topics to be auto created if they do not exist when fetching their metadata. If set to false, the topic must already exist.

Type: bool

Default: true

batching

Configure a batching policy.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

batching.byte_size

The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).

Set to 0 to disable size-based batching. When disabled, messages are flushed based on other conditions (such as count or period).

Type: int

Default: 0

batching.check

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

batching.count

The number of messages at which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

batching.period

The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s. Supported time units are ns, us, ms, s, m, and h.

Type: string

Default: ""

# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms

batching.processors[]

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.

Type: array<processor>

# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array

broker_write_max_bytes

The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s socket.request.max.bytes.

Type: string

Default: 100MiB

# Examples:
broker_write_max_bytes: 128MB

# ---

broker_write_max_bytes: 50mib

client_id

An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting.

Type: string

Default: redpanda-connect

compression

Set an explicit compression type (optional). By default, the client uses snappy when the broker supports it, and falls back to none if not.

Type: string

Options: lz4, snappy, gzip, none, zstd

conn_idle_timeout

The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: 20s

idempotent_write

Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the IDEMPOTENT_WRITE permission on the CLUSTER resource. Disable this option if the IDEMPOTENT_WRITE permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally.

Type: bool

Default: true

key

An optional key to populate for each message.

This field supports interpolation functions.

Type: string

max_buffered_bytes

The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered. Set to 0 to disable the byte-level limit (only max_buffered_records applies). This limit is checked after max_buffered_records.

Type: string

Default: 0

# Examples:
max_buffered_bytes: 256MB

# ---

max_buffered_bytes: 50mib

max_buffered_records

The maximum number of records the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls.

Type: int

Default: 10000

max_in_flight

The maximum number of message batches to send in parallel at any given time.

Type: int

Default: 10

max_in_flight_requests

The maximum number of produce requests in flight per broker connection. While idempotent_write is enabled (the default) this must be 1, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set idempotent_write to false, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s max_in_flight field, which counts message batches being written in parallel rather than produce requests on the wire. A value of 1 is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests.

Type: int

Default: 1

max_message_bytes

The maximum size of a produced record batch in bytes. A MESSAGE_TOO_LARGE error is returned if a batch exceeds this limit. This field maps to the max.message.bytes Kafka property. Ensure the Redpanda broker’s kafka_batch_max_bytes property is at least as large as this value, see kafka_batch_max_bytes.

Type: string

Default: 1MiB

# Examples:
max_message_bytes: 100MB

# ---

max_message_bytes: 50mib

metadata

Determines which metadata values are added to messages as headers.

Type: object

metadata.include_patterns[]

A list of explicit metadata key regular expression (re2) patterns to match against.

Type: array<string>

Default: []

# Examples:
include_patterns:
  - .*

# ---

include_patterns:
  - _timestamp_unix$

metadata.include_prefixes[]

A list of explicit metadata key prefixes to match against.

Type: array<string>

Default: []

# Examples:
include_prefixes:
  - foo_
  - bar_

# ---

include_prefixes:
  - kafka_

# ---

include_prefixes:
  - content-

metadata_max_age

The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes.

This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics.

Type: string

Default: 1m

partition

An optional explicit partition to set for each message. This field is only relevant when the partitioner is set to manual. The provided interpolation string must be a valid integer.

This field supports interpolation functions.

Type: string

# Examples:
partition: ${! meta("partition") }

partitioner

Override the default murmur2 hashing partitioner.

Type: string

Option Summary

least_backup

Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch.

manual

Manually select a partition for each message, requires the field partition to be specified.

murmur2_hash

Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on.

round_robin

Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions.

record_delivery_timeout

The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s delivery.timeout.ms. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to 0s for no timeout (the default). With idempotent_write enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers.

Type: string

Default: 0s

record_retries

The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to 0 for unlimited retries (the default). With idempotent_write enabled, retries are only enforced when safe to do so without creating invalid sequence numbers.

Type: int

Default: 0

request_timeout_overhead

Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts.

This field is roughly equivalent to Apache Kafka’s request.timeout.ms parameter, but grants extra time to requests that have timeout fields.

Type: string

Default: 10s

sasl[]

Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail.

Type: array<object>

# Examples:
sasl:
  - mechanism: SCRAM-SHA-512
    password: bar
    username: foo

sasl[].aws

Contains AWS-specific fields for when sasl.mechanism is set to AWS_MSK_IAM.

Type: object

sasl[].aws.credentials

Manually configure the AWS credentials to use (optional). For more information, see the Amazon Web Services guide.

Type: object

sasl[].aws.credentials.from_ec2_role

Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance.

Type: bool

sasl[].aws.credentials.id

The ID of the AWS credentials to use.

Type: string

sasl[].aws.credentials.profile

The profile from ~/.aws/credentials to use.

Type: string

sasl[].aws.credentials.role

The ARN of the role to assume.

Type: string

sasl[].aws.credentials.role_external_id

An external ID to use when assuming a role.

Type: string

sasl[].aws.credentials.secret

The secret for the AWS credentials in use.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

sasl[].aws.credentials.token

The token for the AWS credentials in use. Required only when using short-term credentials.

Type: string

sasl[].aws.endpoint

A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints.

Type: string

sasl[].aws.region

The AWS region in which your resources are hosted.

Type: string

sasl[].aws.tcp

Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:

  • Unresponsive hosts: Set connect_timeout to limit how long a connection attempt can take (the default 0s sets no limit)

  • Long-lived connections: Configure keep_alive settings to detect and recover from stale connections

  • Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives

  • Linux systems with specific requirements: Use tcp_user_timeout (Linux 2.6.37+) to control data acknowledgment timeouts

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Type: object

sasl[].aws.tcp.connect_timeout

Maximum amount of time a dial will wait for a connect to complete. Zero disables.

Type: string

Default: 0s

sasl[].aws.tcp.keep_alive

TCP keep-alive probe configuration.

Type: object

sasl[].aws.tcp.keep_alive.count

Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.

Type: int

Default: 9

sasl[].aws.tcp.keep_alive.idle

Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.

Type: string

Default: 15s

sasl[].aws.tcp.keep_alive.interval

Duration between keep-alive probes. Zero defaults to 15s.

Type: string

Default: 15s

sasl[].aws.tcp.tcp_user_timeout

Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep_alive.idle must be greater than this value per RFC 5482. Zero disables.

Type: string

Default: 0s

sasl[].extensions

Key/value pairs to add to OAUTHBEARER authentication requests.

Type: object<string>

sasl[].mechanism

The SASL mechanism to use for authentication.

Type: string

Option Summary

AWS_MSK_IAM

AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library.

OAUTHBEARER

OAuth Bearer based authentication.

PLAIN

Plain text authentication.

REDPANDA_CLOUD_SERVICE_ACCOUNT

Redpanda Cloud Service Account authentication when running in Redpanda Cloud.

SCRAM-SHA-256

SCRAM based authentication as specified in RFC5802.

SCRAM-SHA-512

SCRAM based authentication as specified in RFC5802.

none

Disable sasl authentication

sasl[].password

The password to use for PLAIN or SCRAM-* authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

sasl[].token

The token to use for a single session’s OAUTHBEARER authentication.

Type: string

Default: ""

sasl[].username

The username to use for PLAIN or SCRAM-* authentication.

Type: string

Default: ""

seed_brokers[]

A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses.

Type: array<string>

# Examples:
seed_brokers:
  - "localhost:9092"

# ---

seed_brokers:
  - "foo:9092"
  - "bar:9092"

# ---

seed_brokers:
  - "foo:9092,bar:9092"

tcp

Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:

  • Unresponsive hosts: Set connect_timeout to limit how long a connection attempt can take (the default 0s sets no limit)

  • Long-lived connections: Configure keep_alive settings to detect and recover from stale connections

  • Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives

  • Linux systems with specific requirements: Use tcp_user_timeout (Linux 2.6.37+) to control data acknowledgment timeouts

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Type: object

tcp.connect_timeout

Maximum amount of time a dial will wait for a connect to complete. Zero disables.

Type: string

Default: 0s

tcp.keep_alive

TCP keep-alive probe configuration.

Type: object

tcp.keep_alive.count

Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.

Type: int

Default: 9

tcp.keep_alive.idle

Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.

Type: string

Default: 15s

tcp.keep_alive.interval

Duration between keep-alive probes. Zero defaults to 15s.

Type: string

Default: 15s

tcp.tcp_user_timeout

Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep_alive.idle must be greater than this value per RFC 5482. Zero disables.

Type: string

Default: 0s

timeout

The maximum period of time to wait for message sends before abandoning the request and retrying.

Type: string

Default: 10s

timestamp_ms

An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both timestamp and timestamp_ms.

This field supports interpolation functions.

Type: string

# Examples:
timestamp_ms: ${! timestamp_unix_milli() }

# ---

timestamp_ms: ${! metadata("kafka_timestamp_ms") }

tls

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

tls.client_certs[]

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

  • Inline PEM data using both cert and key or

  • File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

tls.client_certs[].cert

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

tls.client_certs[].cert_file

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

tls.client_certs[].key

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

tls.client_certs[].key_file

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

tls.client_certs[].password

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

tls.enable_renegotiation

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

tls.enabled

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

tls.skip_cert_verify

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

topic

A topic to write messages to.

This field supports interpolation functions.

Type: string