Connect
Self-Managed Only

ockam_kafka

Uses Ockam to encrypt and write end-to-end encrypted messages to a Kafka topic. You can read encrypted messages using the ockam_kafka input or by creating a Kafka Portal Inlet using Ockam Command.

Ockam Secure Channels guarantee that consumers of topics only receive data from authenticated and authorized producers, and that the data has not been leaked or tampered with in-flight. Neither Kafka brokers, service providers nor other components can see or manipulate the messages.

You can use Ockam to encrypt whole messages, or specific fields in a message.

Introduced in version 4.33.0.

  • Common

  • Advanced

output:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      max_in_flight: 10
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
      topic: "" # No default (required)
      key: "" # No default (optional)
      partition: "" # No default (optional)
      metadata:
        include_prefixes: []
        include_patterns: []
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_consumer: self
    route_to_consumer: /ip4/127.0.0.1/tcp/6262
    encrypted_fields: []
output:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      max_in_flight: 10
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      partitioner: "" # No default (optional)
      idempotent_write: true
      acks: all
      compression: "" # No default (optional)
      allow_auto_topic_creation: true
      timeout: 10s
      max_message_bytes: 1MiB
      broker_write_max_bytes: 100MiB
      max_buffered_records: 10000
      max_buffered_bytes: "0"
      max_in_flight_requests: 1
      record_retries: 0
      record_delivery_timeout: 0s
      topic: "" # No default (required)
      key: "" # No default (optional)
      partition: "" # No default (optional)
      metadata:
        include_prefixes: []
        include_patterns: []
      timestamp_ms: "" # No default (optional)
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_consumer: self
    route_to_consumer: /ip4/127.0.0.1/tcp/6262
    encrypted_fields: []

Fields

allow

Use in conjunction with the route_to_kafka_outlet field to specify an access control policy for the Kafka Portal Outlet.

For example, setting this value to kafka_us_east forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute kafka_us_east=true.

Type: string

Default: self

allow_consumer

Specify an access control policy for consumers.

For example, setting this value to orders_consumer forces the consumer to present an Ockam credential, which confirms that the consumer has the attribute orders_consumer=true.

Type: string

Default: self

disable_content_encryption

Disables encryption of Kafka message payloads.

If this value is set to true:

  • Only message payloads are unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled.

  • All other ockam_kafka inputs and outputs that use the same topic must also set this field to true.

Type: bool

Default: false

encrypted_fields[]

The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted.

Requires version 4.37.0 or later.

Type: array<string>

Default: []

enrollment_ticket

The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional).

You can generate a new ticket using the ockam project ticket command.

Type: string

identity_name

The name of the Ockam identity to use (optional). If an identity with this name does not exist, it is created. If this value is not provided, the default Ockam identity is used, and it is created if it does not exist.

Type: string

kafka

Type: object

kafka.acks

The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When idempotent_write is enabled this must be set to all.

Requires version 4.92.0 or later.

Type: string

Default: all

Option Summary

all

Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled.

leader

Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication.

none

Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost.

kafka.allow_auto_topic_creation

Enables topics to be auto created if they do not exist when fetching their metadata. If set to false, the topic must already exist.

Requires version 4.58.0 or later.

Type: bool

Default: true

kafka.batching

Configure a batching policy.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

kafka.batching.byte_size

The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).

Set to 0 to disable size-based batching. When disabled, messages are flushed based on other conditions (such as count or period).

Type: int

Default: 0

kafka.batching.check

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

kafka.batching.count

The number of messages at which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

kafka.batching.period

The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s. Supported time units are ns, us, ms, s, m, and h.

Type: string

Default: ""

# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms

kafka.batching.processors[]

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.

Type: array<processor>

# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array

kafka.broker_write_max_bytes

The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s socket.request.max.bytes.

Requires version 4.35.0 or later.

Type: string

Default: 100MiB

# Examples:
broker_write_max_bytes: 128MB

# ---

broker_write_max_bytes: 50mib

kafka.compression

Set an explicit compression type (optional). By default, the client uses snappy when the broker supports it, and falls back to none if not.

Type: string

Options: lz4, snappy, gzip, none, zstd

kafka.idempotent_write

Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the IDEMPOTENT_WRITE permission on the CLUSTER resource. Disable this option if the IDEMPOTENT_WRITE permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally.

Type: bool

Default: true

kafka.key

An optional key to populate for each message.

This field supports interpolation functions.

Type: string

kafka.max_buffered_bytes

The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered. Set to 0 to disable the byte-level limit (only max_buffered_records applies). This limit is checked after max_buffered_records.

Requires version 4.92.0 or later.

Type: string

Default: 0

# Examples:
max_buffered_bytes: 256MB

# ---

max_buffered_bytes: 50mib

kafka.max_buffered_records

The maximum number of records the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls.

Requires version 4.92.0 or later.

Type: int

Default: 10000

kafka.max_in_flight

The maximum number of message batches to send in parallel at any given time.

Type: int

Default: 10

kafka.max_in_flight_requests

The maximum number of produce requests in flight per broker connection. While idempotent_write is enabled (the default) this must be 1, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set idempotent_write to false, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s max_in_flight field, which counts message batches being written in parallel rather than produce requests on the wire. A value of 1 is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests.

Requires version 4.92.0 or later.

Type: int

Default: 1

kafka.max_message_bytes

The maximum size of a produced record batch in bytes. A MESSAGE_TOO_LARGE error is returned if a batch exceeds this limit. This field maps to the max.message.bytes Kafka property. Ensure the Redpanda broker’s kafka_batch_max_bytes property is at least as large as this value, see kafka_batch_max_bytes.

Type: string

Default: 1MiB

# Examples:
max_message_bytes: 100MB

# ---

max_message_bytes: 50mib

kafka.metadata

Determines which metadata values are added to messages as headers.

Type: object

kafka.metadata.include_patterns[]

A list of explicit metadata key regular expression (re2) patterns to match against.

Type: array<string>

Default: []

# Examples:
include_patterns:
  - .*

# ---

include_patterns:
  - _timestamp_unix$

kafka.metadata.include_prefixes[]

A list of explicit metadata key prefixes to match against.

Type: array<string>

Default: []

# Examples:
include_prefixes:
  - foo_
  - bar_

# ---

include_prefixes:
  - kafka_

# ---

include_prefixes:
  - content-

kafka.partition

An optional explicit partition to set for each message. This field is only relevant when the partitioner is set to manual. The provided interpolation string must be a valid integer.

This field supports interpolation functions.

Type: string

# Examples:
partition: ${! meta("partition") }

kafka.partitioner

Override the default murmur2 hashing partitioner.

Type: string

Option Summary

least_backup

Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch.

manual

Manually select a partition for each message, requires the field partition to be specified.

murmur2_hash

Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on.

round_robin

Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions.

kafka.record_delivery_timeout

The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s delivery.timeout.ms. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to 0s for no timeout (the default). With idempotent_write enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers.

Requires version 4.92.0 or later.

Type: string

Default: 0s

kafka.record_retries

The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to 0 for unlimited retries (the default). With idempotent_write enabled, retries are only enforced when safe to do so without creating invalid sequence numbers.

Requires version 4.92.0 or later.

Type: int

Default: 0

kafka.seed_brokers[]

The address of the Kafka broker that the Ockam node’s Kafka Outlet connects to. This field is only used, and is then required, when route_to_kafka_outlet is set to self. Only one address is used: the first list item, up to its first comma. If the list contains more than one item, a warning is logged.

Type: array<string>

# Examples:
seed_brokers:
  - "localhost:9092"

kafka.timeout

The maximum period of time to wait for message sends before abandoning the request and retrying.

Type: string

Default: 10s

kafka.timestamp_ms

An optional timestamp to set for each message, in milliseconds since the Unix epoch. When left empty, the current timestamp is used. You cannot set both timestamp and timestamp_ms.

This field supports interpolation functions.

Requires version 4.40.0 or later.

Type: string

# Examples:
timestamp_ms: ${! timestamp_unix_milli() }

# ---

timestamp_ms: ${! metadata("kafka_timestamp_ms") }

kafka.tls

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

kafka.tls.client_certs[]

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

  • Inline PEM data using both cert and key or

  • File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

kafka.tls.client_certs[].cert

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

kafka.tls.client_certs[].cert_file

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

kafka.tls.client_certs[].key

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

kafka.tls.client_certs[].key_file

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

kafka.tls.client_certs[].password

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

kafka.tls.enable_renegotiation

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

kafka.tls.enabled

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

kafka.tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

kafka.tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

kafka.tls.skip_cert_verify

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

kafka.topic

A topic to write messages to.

This field supports interpolation functions.

Type: string

route_to_consumer

The route to the Kafka consumer. For example, /project/default/service/forward_to_orders_consumer/secure/api connects to a consumer exposed through a relay named orders_consumer.

Type: string

Default: /ip4/127.0.0.1/tcp/6262

route_to_kafka_outlet

The route to reach the Kafka Portal Outlet of your Ockam portal. For example, /project/default.

Type: string

Default: self