Connect

kafka

Deprecated in 4.68.0

This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified redpanda input and redpanda output components.

Connects to Kafka brokers and consumes one or more topics.

Common
Advanced

input:
  label: ""
  kafka:
    addresses: [] # No default (required)
    topics: [] # No default (required)
    target_version: "" # No default (optional)
    consumer_group: ""
    checkpoint_limit: 1024
    auto_replay_nacks: true

input:
  label: ""
  kafka:
    addresses: [] # No default (required)
    topics: [] # No default (required)
    target_version: "" # No default (optional)
    tls:
      enabled: false
      skip_cert_verify: false
      enable_renegotiation: false
      root_cas: ""
      root_cas_file: ""
      client_certs: []
    sasl:
      mechanism: none
      user: ""
      password: ""
      access_token: ""
      token_cache: ""
      token_key: ""
    consumer_group: ""
    client_id: benthos
    instance_id: "" # No default (optional)
    rack_id: ""
    start_from_oldest: true
    checkpoint_limit: 1024
    auto_replay_nacks: true
    timely_nacks_maximum_wait: "" # No default (optional)
    commit_period: 1s
    max_processing_period: 100ms
    extract_tracing_map: "" # No default (optional)
    group:
      session_timeout: 10s
      heartbeat_interval: 3s
      rebalance_timeout: 60s
    fetch_buffer_cap: 256
    multi_header: false
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
      processors: [] # No default (optional)

Offsets are managed within Kafka under the specified consumer group, and partitions for each topic are automatically balanced across members of the consumer group.

The Kafka input allows parallel processing of messages from different topic partitions, and messages of the same topic partition are processed with a maximum parallelism determined by the field checkpoint_limit.

To enforce ordered processing of partition messages, set the checkpoint_limit to 1, which makes sure that a message is only processed after the previous message is delivered.

Batching messages before processing can be enabled using the batching field, and this batching is performed per-partition such that messages of a batch will always originate from the same partition. This batching mechanism is capable of creating batches of greater size than the checkpoint_limit, in which case the next batch will only be created upon delivery of the current one.

Metadata

This input adds the following metadata fields to each message:

kafka_key
kafka_topic
kafka_partition
kafka_offset
kafka_lag
kafka_timestamp_ms
kafka_timestamp_unix
kafka_tombstone_message
All existing message headers (version 0.11+)

The field kafka_lag is the calculated difference between the high water mark offset of the partition at the time of ingestion and the current message offset.

You can access these metadata fields using function interpolation.

Ordering

By default messages of a topic partition can be processed in parallel, up to a limit determined by the field checkpoint_limit. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level.

Troubleshooting

If you’re seeing issues writing to or reading from Kafka with this component then it’s worth trying out the newer kafka_franz input.

I’m seeing logs that report Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?), but the brokers are definitely reachable.

Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have enabled TLS if applicable.

Fields

`addresses[]`

A list of broker addresses to connect to. List items that contain commas are expanded into multiple addresses.

Type: array<string>

# Examples:
addresses:
  - "localhost:9092"

# ---

addresses:
  - "localhost:9041,localhost:9042"

# ---

addresses:
  - "localhost:9041"
  - "localhost:9042"

`auto_replay_nacks`

Whether to automatically replay rejected messages (negative acknowledgements, or nacks) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure.

Set auto_replay_nacks to false to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.

Requires version 4.27.0 or later.

Type: bool

Default: true

`batching`

Configure a batching policy.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

`batching.byte_size`

The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).

Set to 0 to disable size-based batching. When disabled, messages are flushed based on other conditions (such as count or period).

Type: int

Default: 0

`batching.check`

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

`batching.count`

The number of messages at which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

`batching.period`

The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s. Supported time units are ns, us, ms, s, m, and h.

Type: string

Default: ""

# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms

`batching.processors[]`

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.

Type: array<processor>

# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array

`checkpoint_limit`

The maximum number of messages of the same topic and partition that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual partitions. Any given offset will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees.

Requires version 3.33.0 or later.

Type: int

Default: 1024

`client_id`

The client ID sent to brokers with every request, which lets brokers attribute requests to this client in their logs, metrics, and client quotas.

Type: string

Default: benthos

`commit_period`

The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown.

Type: string

Default: 1s

`consumer_group`

An identifier for the consumer group of the connection. This field can be explicitly made empty in order to disable stored offsets for the consumed topic partitions.

Type: string

Default: ""

`extract_tracing_map`

EXPERIMENTAL: A Bloblang mapping that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer.

Requires version 3.45.0 or later.

Type: string

# Examples:
extract_tracing_map: root = @

# ---

extract_tracing_map: root = this.meta.span

`fetch_buffer_cap`

The maximum number of unprocessed messages to fetch at a given time.

Type: int

Default: 256

`group`

Tuning parameters for consumer group synchronization.

Type: object

`group.heartbeat_interval`

A period in which heartbeats should be sent out.

Type: string

Default: 3s

`group.rebalance_timeout`

A period after which rebalancing is abandoned if unresolved.

Type: string

Default: 60s

`group.session_timeout`

A period after which a consumer of the group is kicked after no heartbeats.

Type: string

Default: 10s

`instance_id`

When you specify a consumer_group, assign a unique value to instance_id for each input so that brokers can identify it across restarts of this process and avoid unnecessary rebalances.

Requires version 4.46.0 or later.

Type: string

`max_processing_period`

A maximum estimate for the time taken to process a message, this is used for tuning consumer group synchronization.

Type: string

Default: 100ms

`multi_header`

Decode headers into lists to allow handling of multiple values with the same key

Requires version 4.10.0 or later.

Type: bool

Default: false

`rack_id`

A rack identifier for this client.

Type: string

Default: ""

`sasl`

Enables SASL authentication.

Type: object

`sasl.access_token`

A static OAUTHBEARER access token

Type: string

Default: ""

`sasl.mechanism`

The SASL authentication mechanism, if left empty SASL authentication is not used.

Type: string

Default: none

Option Summary

Option	Summary
`OAUTHBEARER`	OAuth Bearer based authentication.
`PLAIN`	Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS.
`SCRAM-SHA-256`	Authentication using the SCRAM-SHA-256 mechanism.
`SCRAM-SHA-512`	Authentication using the SCRAM-SHA-512 mechanism.
`none`	Default, no SASL authentication.

OAUTHBEARER

OAuth Bearer based authentication.

PLAIN

Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS.

SCRAM-SHA-256

Authentication using the SCRAM-SHA-256 mechanism.

SCRAM-SHA-512

Authentication using the SCRAM-SHA-512 mechanism.

none

Default, no SASL authentication.

`sasl.password`

A PLAIN password. It is recommended that you use environment variables to populate this field.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
password: ${PASSWORD}

`sasl.token_cache`

Instead of using a static access_token allows you to query a cache resource to fetch OAUTHBEARER tokens from

Type: string

Default: ""

`sasl.token_key`

Required when using a token_cache, the key to query the cache with for tokens.

Type: string

Default: ""

`sasl.user`

A PLAIN username. It is recommended that you use environment variables to populate this field.

Type: string

Default: ""

# Examples:
user: ${USER}

`start_from_oldest`

Determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. The setting is applied when creating a new consumer group or the saved offset no longer exists.

Type: bool

Default: true

`target_version`

The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version.

Type: string

# Examples:
target_version: 2.1.0

# ---

target_version: 3.1.0

`timely_nacks_maximum_wait`

EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as 100ms, 1s, or 5s.

Requires version 4.62.0 or later.

Type: string

`tls`

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

`tls.client_certs[]`

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

Inline PEM data using both cert and key or
File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

`tls.client_certs[].cert`

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

`tls.client_certs[].cert_file`

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

`tls.client_certs[].key`

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`tls.client_certs[].key_file`

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

`tls.client_certs[].password`

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Requires version 4.3.0 or later.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

`tls.enable_renegotiation`

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Requires version 3.45.0 or later.

Type: bool

Default: false

`tls.enabled`

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

`tls.root_cas`

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

`tls.root_cas_file`

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

`tls.skip_cert_verify`

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

`topics[]`

A list of topics to consume from. Multiple comma separated topics can be listed in a single element. Partitions are automatically distributed across consumers of a topic. Alternatively, it’s possible to specify explicit partitions to consume from with a colon after the topic name. For example foo:0 would consume the partition 0 of the topic foo. This syntax supports ranges. For example foo:0-10 would consume partitions 0 through to 10 inclusive.

Requires version 3.33.0 or later.

Type: array<string>

# Examples:
topics:
  - foo
  - bar

# ---

topics:
  - "foo,bar"

# ---

topics:
  - "foo:0"
  - "bar:1"
  - "bar:3"

# ---

topics:
  - "foo:0,bar:1,bar:3"

# ---

topics:
  - "foo:0-5"

Was this helpful?

group Ask in the community

mail Share your feedback

What do you think of this page?

Let us know more:

Let us contact you about your feedback:

kafka

Metadata

Ordering

Troubleshooting

Fields

addresses[]

auto_replay_nacks

batching

batching.byte_size

batching.check

batching.count

batching.period

batching.processors[]

checkpoint_limit

client_id

commit_period

consumer_group

extract_tracing_map

fetch_buffer_cap

group

group.heartbeat_interval

group.rebalance_timeout

group.session_timeout

instance_id

max_processing_period

multi_header

rack_id

sasl

sasl.access_token

sasl.mechanism

sasl.password

sasl.token_cache

sasl.token_key

sasl.user

start_from_oldest

target_version

timely_nacks_maximum_wait

tls

tls.client_certs[]

tls.client_certs[].cert

tls.client_certs[].cert_file

tls.client_certs[].key

tls.client_certs[].key_file

tls.client_certs[].password

tls.enable_renegotiation

tls.enabled

tls.root_cas

tls.root_cas_file

tls.skip_cert_verify

topics[]

`addresses[]`

`auto_replay_nacks`

`batching`

`batching.byte_size`

`batching.check`

`batching.count`

`batching.period`

`batching.processors[]`

`checkpoint_limit`

`client_id`

`commit_period`

`consumer_group`

`extract_tracing_map`

`fetch_buffer_cap`

`group`

`group.heartbeat_interval`

`group.rebalance_timeout`

`group.session_timeout`

`instance_id`

`max_processing_period`

`multi_header`

`rack_id`

`sasl`

`sasl.access_token`

`sasl.mechanism`

`sasl.password`

`sasl.token_cache`

`sasl.token_key`

`sasl.user`

`start_from_oldest`

`target_version`

`timely_nacks_maximum_wait`

`tls`

`tls.client_certs[]`

`tls.client_certs[].cert`

`tls.client_certs[].cert_file`

`tls.client_certs[].key`

`tls.client_certs[].key_file`

`tls.client_certs[].password`

`tls.enable_renegotiation`

`tls.enabled`

`tls.root_cas`

`tls.root_cas_file`

`tls.skip_cert_verify`

`topics[]`