Connect
Self-Managed Only

ockam_kafka

Uses Ockam to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the ockam_kafka output or by creating a Kafka Portal Inlet using Ockam Command.

Ockam Secure Channels guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages.

You can use Ockam to encrypt whole messages, or specific fields in a message.

Introduced in version 4.33.0.

  • Common

  • Advanced

input:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      topics: [] # No default (optional)
      regexp_topics_include: [] # No default (optional)
      regexp_topics_exclude: [] # No default (optional)
      transaction_isolation_level: read_uncommitted
      consumer_group: "" # No default (optional)
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
    encrypted_fields: []
input:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      topics: [] # No default (optional)
      regexp_topics_include: [] # No default (optional)
      regexp_topics_exclude: [] # No default (optional)
      rack_id: ""
      instance_id: ""
      rebalance_timeout: 45s
      session_timeout: 1m
      heartbeat_interval: 3s
      start_offset: earliest
      fetch_max_bytes: 50MiB
      fetch_max_wait: 5s
      fetch_min_bytes: 1B
      fetch_max_partition_bytes: 1MiB
      transaction_isolation_level: read_uncommitted
      consumer_group: "" # No default (optional)
      checkpoint_limit: 1024
      commit_period: 5s
      multi_header: false
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      topic_lag_refresh_period: 5s
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
    encrypted_fields: []

Fields

allow

Use in conjunction with the route_to_kafka_outlet field to specify an access control policy for the Kafka Portal Outlet.

For example, setting this value to kafka_us_east forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute kafka_us_east=true.

Type: string

Default: self

allow_producer

Specify an access control policy for producers.

For example, setting this value to orders_producer forces the producer to present an Ockam credential, which confirms that the producer has the attribute orders_producer=true.

Type: string

Default: self

disable_content_encryption

Disables encryption of Kafka message payloads.

If this value is set to true:

  • Only message payloads are unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled.

  • All other ockam_kafka inputs and outputs that use the same topic must also set this field to true.

Type: bool

Default: false

encrypted_fields[]

The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted.

Requires version 4.37.0 or later.

Type: array<string>

Default: []

enrollment_ticket

The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional).

You can generate a new ticket using the ockam project ticket command.

Type: string

identity_name

The name of the Ockam identity to use (optional). If an identity with this name does not exist, it is created. If this value is not provided, the default Ockam identity is used, and it is created if it does not exist.

Type: string

kafka

Type: object

kafka.batching

Configure a batching policy that applies to individual topic partitions, so that the input batches messages together before flushing them for processing. Batching can improve performance and is useful for windowed processing. Batching this way preserves the ordering of topic partitions.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

kafka.batching.byte_size

The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).

Set to 0 to disable size-based batching. When disabled, messages are flushed based on other conditions (such as count or period).

Type: int

Default: 0

kafka.batching.check

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

kafka.batching.count

The number of messages at which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

kafka.batching.period

The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s. Supported time units are ns, us, ms, s, m, and h.

Type: string

Default: ""

# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms

kafka.batching.processors[]

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.

Type: array<processor>

# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array

kafka.checkpoint_limit

The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied.

When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the checkpoint_limit value.

Type: int

Default: 1024

kafka.commit_period

The period of time between each commit of the current partition offsets to the consumer group. Offsets are always committed during shutdown.

Type: string

Default: 5s

kafka.consumer_group

An optional consumer group. When you specify this value:

  • The partitions of any topics specified in the topics field are automatically distributed across consumers that share the consumer group.

  • Partition offsets are automatically committed and resumed under this name.

Consumer groups are not supported when you specify explicit partitions to consume from in the topics field.

Type: string

kafka.fetch_max_bytes

The maximum number of bytes that a broker tries to send during a fetch.

If individual records are larger than the fetch_max_bytes value, brokers still send them.

This field is equivalent to the Java setting fetch.max.bytes.

Requires version 4.39.0 or later.

Type: string

Default: 50MiB

kafka.fetch_max_partition_bytes

The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting fetch.max.partition.bytes.

If a single batch is larger than the fetch_max_partition_bytes value, the batch is still sent so that the client can make progress.

Requires version 4.39.0 or later.

Type: string

Default: 1MiB

kafka.fetch_max_wait

The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (fetch_min_bytes). This field is equivalent to the Java setting fetch.max.wait.ms.

Requires version 4.45.0 or later.

Type: string

Default: 5s

kafka.fetch_min_bytes

The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting fetch.min.bytes.

Requires version 4.39.0 or later.

Type: string

Default: 1B

kafka.heartbeat_interval

When you specify a consumer_group, heartbeat_interval sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active.

This value must be lower than session_timeout, and should be no higher than one-third of session_timeout.

This field is equivalent to the Java heartbeat.interval.ms setting and accepts Go duration format strings such as 10s or 2m.

Requires version 4.46.0 or later.

Type: string

Default: 3s

kafka.instance_id

When you specify a consumer_group, set instance_id to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. The value must be unique per consumer within the group.

When you assign an instance ID, the client does not leave the consumer group when it closes. To remove the client from the group, you must use an external admin command on behalf of the instance ID.

Requires version 4.46.0 or later.

Type: string

Default: ""

kafka.multi_header

Decode headers into lists to allow the handling of multiple values with the same key.

Type: bool

Default: false

kafka.rack_id

A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica.

Type: string

Default: ""

kafka.rebalance_timeout

When you specify a consumer_group, rebalance_timeout sets how long consumer group members can take to complete their work and commit offsets after a rebalance has begun. The time that a member takes to detect the rebalance (from a heartbeat) counts against this timeout. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Requires version 4.46.0 or later.

Type: string

Default: 45s

kafka.regexp_topics_exclude[]

A list of regular expression patterns for excluding topics when regex mode is enabled (using regexp_topics_include or the deprecated regexp_topics boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns.

Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use ^ and $ for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection.

Example: regexp_topics_exclude: ["^_", ".-temp$", ".-test.*"] excludes topics starting with underscore, ending with -temp, or containing -test.

Requires version 4.69.0 or later.

Type: array<string>

kafka.regexp_topics_include[]

A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the metadata_max_age interval.

Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so logs_. matches my-logs_events and logs_errors. Use ^logs_.$ to match only topics starting with logs_.

This field enables regex mode (replacing the deprecated regexp_topics boolean) and cannot be used together with explicit topics lists. Use regexp_topics_exclude to filter out specific patterns from the matched topics.

Example: regexp_topics_include: ["events_.", "logs_."] consumes from all topics starting with events_ or logs_.

Requires version 4.69.0 or later.

Type: array<string>

# Examples:
regexp_topics_include:
  - logs_.*
  - metrics_.*

# ---

regexp_topics_include:
  - "events_[0-9]+"

kafka.seed_brokers[]

The address of the Kafka broker that the Ockam node’s Kafka Outlet connects to. This field is only used, and is then required, when route_to_kafka_outlet is set to self. Only one address is used: the first list item, up to its first comma. If the list contains more than one item, a warning is logged.

Type: array<string>

# Examples:
seed_brokers:
  - "localhost:9092"

kafka.session_timeout

When you specify a consumer_group, session_timeout sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Requires version 4.46.0 or later.

Type: string

Default: 1m

kafka.start_offset

Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the OffsetOutOfRange error is seen during a fetch.

Requires version 4.53.0 or later.

Type: string

Default: earliest

Option Summary

committed

Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option

earliest

Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option.

latest

Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option.

kafka.tls

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

kafka.tls.client_certs[]

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

  • Inline PEM data using both cert and key or

  • File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

kafka.tls.client_certs[].cert

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

kafka.tls.client_certs[].cert_file

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

kafka.tls.client_certs[].key

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

kafka.tls.client_certs[].key_file

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

kafka.tls.client_certs[].password

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

kafka.tls.enable_renegotiation

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

kafka.tls.enabled

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

kafka.tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

kafka.tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

kafka.tls.skip_cert_verify

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

kafka.topic_lag_refresh_period

The interval between consumer lag refreshes. During each cycle, this input asks the brokers for the consumer group’s committed offsets and the partition end offsets, and records the difference (the number of unread messages) for each topic partition in the consumer lag metric and the kafka_lag metadata field. Lag is only refreshed when consumer_group is set. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Requires version 4.54.1 or later.

Type: string

Default: 5s

kafka.topics[]

A list of topics to consume from. You can list multiple comma-separated topics in a single element.

If you specify a consumer_group, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed.

Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, foo:0 consumes the partition 0 of the topic foo. This syntax also supports ranges. For example, foo:0-10 consumes all partitions from 0 through to 10 inclusive.

Finally, add another colon after the partition to set an explicit offset to consume from. For example, foo:0:10 consumes the partition 0 of the topic foo starting from the offset 10. If the offset is not present (or remains unspecified) then the field start_offset determines which offset to start from.

Type: array<string>

# Examples:
topics:
  - foo
  - bar

# ---

topics:
  - things.*

# ---

topics:
  - "foo,bar"

# ---

topics:
  - "foo:0"
  - "bar:1"
  - "bar:3"

# ---

topics:
  - "foo:0,bar:1,bar:3"

# ---

topics:
  - "foo:0-5"

kafka.transaction_isolation_level

The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees.

Requires version 4.53.0 or later.

Type: string

Default: read_uncommitted

Option Summary

read_committed

If set, only committed transactional records are processed.

read_uncommitted

If set, then uncommitted records are processed.

node_address

The TCP listening address of the Ockam node.

Type: string

Default: 127.0.0.1:6262

relay

Make the Ockam node accessible through a relay with the supplied name (optional). This field only takes effect when enrollment_ticket is set.

For example, setting this value to orders_consumer requires you to set route_to_consumer on any producer to /project/default/service/forward_to_orders_consumer/secure/api.

Type: string

route_to_kafka_outlet

The route to reach the Kafka Portal Outlet of your Ockam portal. For example, /project/default.

Type: string

Default: self