Connect

redpanda

Export distributed tracing data to a Redpanda topic, enabling you to monitor and debug your Redpanda Connect pipelines. Traces are exported in OpenTelemetry format as JSON, allowing integration with observability platforms like Jaeger, Grafana Tempo, or custom trace consumers.

Common
Advanced

tracer:
  redpanda:
    seed_brokers: [] # No default (required)
    topic: otel-traces
    format: json
    schema_registry:
      url: "" # No default (optional)
    service: redpanda-connect
    sampling:
      enabled: false
      ratio: 0 # No default (optional)

tracer:
  redpanda:
    seed_brokers: [] # No default (required)
    client_id: redpanda-connect
    tls:
      enabled: false
      skip_cert_verify: false
      enable_renegotiation: false
      root_cas: ""
      root_cas_file: ""
      client_certs: []
    sasl: [] # No default (optional)
    metadata_max_age: 1m
    request_timeout_overhead: 10s
    conn_idle_timeout: 20s
    tcp:
      connect_timeout: 0s
      keep_alive:
        idle: 15s
        interval: 15s
        count: 9
      tcp_user_timeout: 0s
    partitioner: "" # No default (optional)
    idempotent_write: true
    acks: all
    compression: "" # No default (optional)
    allow_auto_topic_creation: true
    timeout: 10s
    max_message_bytes: 1MiB
    broker_write_max_bytes: 100MiB
    max_buffered_records: 10000
    max_buffered_bytes: "0"
    max_in_flight_requests: 1
    record_retries: 0
    record_delivery_timeout: 0s
    topic: otel-traces
    format: json
    schema_registry:
      url: "" # No default (optional)
      tls:
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      oauth2:
        enabled: false
        client_key: ""
        client_secret: ""
        token_url: ""
        scopes: []
        endpoint_params: {}
      oauth:
        enabled: false
        consumer_key: ""
        consumer_secret: ""
        access_token: ""
        access_token_secret: ""
      basic_auth:
        enabled: false
        username: ""
        password: ""
      jwt:
        enabled: false
        private_key_file: ""
        signing_method: ""
        claims: {}
        headers: {}
    service: redpanda-connect
    tags: {}
    sampling:
      enabled: false
      ratio: 0 # No default (optional)

This tracer automatically captures trace spans as messages flow through your pipeline, recording timing information, component metadata, and error details. Use this to:

Track message flow through complex pipelines with multiple processors.
Identify performance bottlenecks by analyzing span durations.
Debug failures by examining trace context and error details.
Monitor pipeline health across distributed Redpanda Connect instances.
Correlate activity across multiple services using trace IDs.

The tracer writes to a dedicated Redpanda topic that can be consumed by trace analysis tools. Configure sampling to control trace volume in high-throughput environments.

Fields

`acks`

The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When idempotent_write is enabled this must be set to all.

Requires version 4.92.0 or later.

Type: string

Default: all

Option Summary

Option	Summary
`all`	Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled.
`leader`	Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication.
`none`	Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost.

all

Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled.

leader

Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication.

none

Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost.

`allow_auto_topic_creation`

Enables topics to be auto created if they do not exist when fetching their metadata. If set to false, the topic must already exist.

Type: bool

Default: true

`broker_write_max_bytes`

The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s socket.request.max.bytes.

Type: string

Default: 100MiB

# Examples:
broker_write_max_bytes: 128MB

# ---

broker_write_max_bytes: 50mib

`client_id`

An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting.

Type: string

Default: redpanda-connect

`compression`

Set an explicit compression type (optional). By default, the client uses snappy when the broker supports it, and falls back to none if not.

Type: string

Options: lz4, snappy, gzip, none, zstd

`conn_idle_timeout`

The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: 20s

`format`

The serialization format for individual spans in the topic.

Type: string

Default: json

Option Summary

Option	Summary
`json`	Emit in JSON Format
`protobuf`	Emit in Protobuf Format
`schema-registry-json`	Emit in JSON Format with Schema Registry encoding
`schema-registry-protobuf`	Emit in Protobuf Format with Schema Registry encoding

json

Emit in JSON Format

protobuf

Emit in Protobuf Format

schema-registry-json

Emit in JSON Format with Schema Registry encoding

schema-registry-protobuf

Emit in Protobuf Format with Schema Registry encoding

`idempotent_write`

Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the IDEMPOTENT_WRITE permission on the CLUSTER resource. Disable this option if the IDEMPOTENT_WRITE permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally.

Type: bool

Default: true

`max_buffered_bytes`

The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered. Set to 0 to disable the byte-level limit (only max_buffered_records applies). This limit is checked after max_buffered_records.

Requires version 4.92.0 or later.

Type: string

Default: 0

# Examples:
max_buffered_bytes: 256MB

# ---

max_buffered_bytes: 50mib

`max_buffered_records`

The maximum number of records the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls.

Requires version 4.92.0 or later.

Type: int

Default: 10000

`max_in_flight_requests`

The maximum number of produce requests in flight per broker connection. While idempotent_write is enabled (the default) this must be 1, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set idempotent_write to false, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s max_in_flight field, which counts message batches being written in parallel rather than produce requests on the wire. A value of 1 is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests.

Requires version 4.92.0 or later.

Type: int

Default: 1

`max_message_bytes`

The maximum size of a produced record batch in bytes. A MESSAGE_TOO_LARGE error is returned if a batch exceeds this limit. This field maps to the max.message.bytes Kafka property. Ensure the Redpanda broker’s kafka_batch_max_bytes property is at least as large as this value, see kafka_batch_max_bytes.

Type: string

Default: 1MiB

# Examples:
max_message_bytes: 100MB

# ---

max_message_bytes: 50mib

`metadata_max_age`

The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes.

This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics.

Type: string

Default: 1m

`partitioner`

Override the default murmur2 hashing partitioner.

Type: string

Option Summary

Option	Summary
`least_backup`	Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch.
`manual`	Manually select a partition for each message, requires the field `partition` to be specified.
`murmur2_hash`	Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on.
`round_robin`	Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions.

least_backup

Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch.

manual

Manually select a partition for each message, requires the field partition to be specified.

murmur2_hash

Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on.

round_robin

Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions.

`record_delivery_timeout`

The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s delivery.timeout.ms. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to 0s for no timeout (the default). With idempotent_write enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers.

Requires version 4.92.0 or later.

Type: string

Default: 0s

`record_retries`

The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to 0 for unlimited retries (the default). With idempotent_write enabled, retries are only enforced when safe to do so without creating invalid sequence numbers.

Requires version 4.92.0 or later.

Type: int

Default: 0

`request_timeout_overhead`

Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts.

This field is roughly equivalent to Apache Kafka’s request.timeout.ms parameter, but grants extra time to requests that have timeout fields.

Type: string

Default: 10s

`sampling`

Configure trace sampling to control the volume of trace data. Sampling is recommended for high-volume production workloads to prevent trace data from overwhelming your observability infrastructure.

Type: object

`sampling.enabled`

Whether to enable trace sampling. When disabled, all traces are exported. When enabled, traces are sampled according to the configured ratio.

Type: bool

Default: false

`sampling.ratio`

The sampling ratio as a decimal between 0 and 1. For example, 0.1 samples 10% of traces, 0.01 samples 1%. Lower ratios reduce trace volume and overhead. For high-throughput production systems, start with 0.01-0.1 and adjust based on your needs.

Type: float

# Examples:
ratio: 0.05

# ---

ratio: 0.85

# ---

ratio: 0.5

`sasl[]`

Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail.

Type: array<object>

# Examples:
sasl:
  - mechanism: SCRAM-SHA-512
    password: bar
    username: foo

`sasl[].aws`

Contains AWS-specific fields for when sasl.mechanism is set to AWS_MSK_IAM.

Type: object

`sasl[].aws.credentials`

Manually configure the AWS credentials to use (optional). For more information, see the Amazon Web Services guide.

Type: object

`sasl[].aws.credentials.from_ec2_role`

Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance.

Type: bool

`sasl[].aws.credentials.id`

The ID of the AWS credentials to use.

Type: string

`sasl[].aws.credentials.profile`

The profile from ~/.aws/credentials to use.

Type: string

`sasl[].aws.credentials.role`

The ARN of the role to assume.

Type: string

`sasl[].aws.credentials.role_external_id`

An external ID to use when assuming a role.

Type: string

`sasl[].aws.credentials.secret`

The secret for the AWS credentials in use.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

`sasl[].aws.credentials.token`

The token for the AWS credentials in use. Required only when using short-term credentials.

Type: string

`sasl[].aws.endpoint`

A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints.

Type: string

`sasl[].aws.region`

The AWS region in which your resources are hosted.

Type: string

`sasl[].aws.tcp`

Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:

Unresponsive hosts: Set connect_timeout to limit how long a connection attempt can take (the default 0s sets no limit)
Long-lived connections: Configure keep_alive settings to detect and recover from stale connections
Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives
Linux systems with specific requirements: Use tcp_user_timeout (Linux 2.6.37+) to control data acknowledgment timeouts

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Type: object

`sasl[].aws.tcp.connect_timeout`

Maximum amount of time a dial will wait for a connect to complete. Zero disables.

Type: string

Default: 0s

`sasl[].aws.tcp.keep_alive`

TCP keep-alive probe configuration.

Type: object

`sasl[].aws.tcp.keep_alive.count`

Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.

Type: int

Default: 9

`sasl[].aws.tcp.keep_alive.idle`

Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.

Type: string

Default: 15s

`sasl[].aws.tcp.keep_alive.interval`

Duration between keep-alive probes. Zero defaults to 15s.

Type: string

Default: 15s

`sasl[].aws.tcp.tcp_user_timeout`

Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep_alive.idle must be greater than this value per RFC 5482. Zero disables.

Type: string

Default: 0s

`sasl[].extensions`

Key/value pairs to add to OAUTHBEARER authentication requests.

Type: object<string>

`sasl[].mechanism`

The SASL mechanism to use for authentication.

Type: string

Option Summary

Option	Summary
`AWS_MSK_IAM`	AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library.
`OAUTHBEARER`	OAuth Bearer based authentication.
`PLAIN`	Plain text authentication.
`REDPANDA_CLOUD_SERVICE_ACCOUNT`	Redpanda Cloud Service Account authentication when running in Redpanda Cloud.
`SCRAM-SHA-256`	SCRAM based authentication as specified in RFC5802.
`SCRAM-SHA-512`	SCRAM based authentication as specified in RFC5802.
`none`	Disable sasl authentication

AWS_MSK_IAM

AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library.

OAUTHBEARER

OAuth Bearer based authentication.

PLAIN

Plain text authentication.

REDPANDA_CLOUD_SERVICE_ACCOUNT

Redpanda Cloud Service Account authentication when running in Redpanda Cloud.

SCRAM-SHA-256

SCRAM based authentication as specified in RFC5802.

SCRAM-SHA-512

SCRAM based authentication as specified in RFC5802.

none

Disable sasl authentication

`sasl[].password`

The password to use for PLAIN or SCRAM-* authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`sasl[].token`

The token to use for a single session’s OAUTHBEARER authentication.

Type: string

Default: ""

`sasl[].username`

The username to use for PLAIN or SCRAM-* authentication.

Type: string

Default: ""

`schema_registry`

Schema registry information to publish schemas for tracing data along with the data.

Requires version 4.73.0 or later.

Type: object

`schema_registry.basic_auth`

Configure basic authentication for requests from this component.

Type: object

`schema_registry.basic_auth.enabled`

Whether to use basic authentication in requests.

Type: bool

Default: false

`schema_registry.basic_auth.password`

The password to use for authentication. Used together with username for basic authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`schema_registry.basic_auth.username`

The username of the account credentials to authenticate as. Used together with password for basic authentication.

Type: string

Default: ""

`schema_registry.jwt`

Beta

Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services.

Type: object

`schema_registry.jwt.claims`

A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider.

Type: object

Default: {}

`schema_registry.jwt.enabled`

Whether to use JWT authentication in requests.

Type: bool

Default: false

`schema_registry.jwt.headers`

Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing.

Type: object

Default: {}

`schema_registry.jwt.private_key_file`

Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the signing_method field.

Type: string

Default: ""

`schema_registry.jwt.signing_method`

The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the private_key_file field.

Type: string

Default: ""

`schema_registry.oauth`

Configure OAuth version 1.0 authentication for secure API access.

Type: object

`schema_registry.oauth.access_token`

The value used to gain access to the protected resources on behalf of the user.

Type: string

Default: ""

`schema_registry.oauth.access_token_secret`

The secret that establishes ownership of the access_token in OAuth 1.0 authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`schema_registry.oauth.consumer_key`

The value used to identify this component or client to the service provider.

Type: string

Default: ""

`schema_registry.oauth.consumer_secret`

The secret that establishes ownership of the consumer key in OAuth 1.0 authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`schema_registry.oauth.enabled`

Whether to enable OAuth version 1.0 authentication for requests.

Type: bool

Default: false

`schema_registry.oauth2`

Allows you to specify open authentication via OAuth version 2 using the client credentials token flow.

Requires version 4.74.0 or later.

Type: object

`schema_registry.oauth2.client_key`

A value used to identify the client to the token provider.

Type: string

Default: ""

`schema_registry.oauth2.client_secret`

A secret used to establish ownership of the client key.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`schema_registry.oauth2.enabled`

Whether to use OAuth version 2 in requests.

Type: bool

Default: false

`schema_registry.oauth2.endpoint_params`

A list of optional endpoint parameters, values should be arrays of strings.

Type: object

Default: {}

# Examples:
endpoint_params:
  audience:
    - https://example.com
  resource:
    - https://api.example.com

`schema_registry.oauth2.scopes[]`

A list of optional requested permissions.

Type: array<string>

Default: []

`schema_registry.oauth2.token_url`

The URL of the token provider.

Type: string

Default: ""

`schema_registry.tls`

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

`schema_registry.tls.client_certs[]`

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

Certificate pairing rules: For each certificate item, provide either:

Inline PEM data using both cert and key or
File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

`schema_registry.tls.client_certs[].cert`

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

`schema_registry.tls.client_certs[].cert_file`

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

`schema_registry.tls.client_certs[].key`

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`schema_registry.tls.client_certs[].key_file`

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

`schema_registry.tls.client_certs[].password`

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

`schema_registry.tls.enable_renegotiation`

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

`schema_registry.tls.root_cas`

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

`schema_registry.tls.root_cas_file`

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

`schema_registry.tls.skip_cert_verify`

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

`schema_registry.url`

The base URL of the schema registry service.

Type: string

`seed_brokers[]`

A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses.

Type: array<string>

# Examples:
seed_brokers:
  - "localhost:9092"

# ---

seed_brokers:
  - "foo:9092"
  - "bar:9092"

# ---

seed_brokers:
  - "foo:9092,bar:9092"

`service`

The service name to identify this Redpanda Connect instance in traces. This appears in trace visualizations and helps correlate traces across distributed systems. Use descriptive names like order-processor or analytics-pipeline.

Type: string

Default: redpanda-connect

`tags`

Custom key-value tags to attach to all traces from this instance. Use tags to add metadata like environment (production, staging), region, version, or instance identifiers. Tags appear as resource attributes in OpenTelemetry traces.

Type: object<string>

Default: {}

`tcp`

Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:

Unresponsive hosts: Set connect_timeout to limit how long a connection attempt can take (the default 0s sets no limit)
Long-lived connections: Configure keep_alive settings to detect and recover from stale connections
Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives
Linux systems with specific requirements: Use tcp_user_timeout (Linux 2.6.37+) to control data acknowledgment timeouts

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Type: object

`tcp.connect_timeout`

Maximum amount of time a dial will wait for a connect to complete. Zero disables.

Type: string

Default: 0s

`tcp.keep_alive`

TCP keep-alive probe configuration.

Type: object

`tcp.keep_alive.count`

Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.

Type: int

Default: 9

`tcp.keep_alive.idle`

Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.

Type: string

Default: 15s

`tcp.keep_alive.interval`

Duration between keep-alive probes. Zero defaults to 15s.

Type: string

Default: 15s

`tcp.tcp_user_timeout`

Type: string

Default: 0s

`timeout`

The maximum period of time to wait for message sends before abandoning the request and retrying.

Type: string

Default: 10s

`tls`

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

`tls.client_certs[]`

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

Inline PEM data using both cert and key or
File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

`tls.client_certs[].cert`

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

`tls.client_certs[].cert_file`

Type: string

Default: ""

`tls.client_certs[].key`

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

`tls.client_certs[].key_file`

Type: string

Default: ""

`tls.client_certs[].password`

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

`tls.enable_renegotiation`

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

`tls.enabled`

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

`tls.root_cas`

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

`tls.root_cas_file`

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

`tls.skip_cert_verify`

Type: bool

Default: false

`topic`

The Redpanda topic where trace data is written. This topic should be dedicated to traces and configured with appropriate retention policies. Default: otel-traces

Type: string

Default: otel-traces

Examples

Basic tracing setup

Enable tracing for a pipeline, sending all trace data to a local Redpanda cluster.

input:
  redpanda:
    seed_brokers: [ "localhost:9092" ]
    topics: [ "orders" ]
    consumer_group: order-processor

pipeline:
  processors:
    - mapping: |
        root = this
        root.processed_at = now()

output:
  redpanda:
    seed_brokers: [ "localhost:9092" ]
    topic: processed_orders

tracer:
  redpanda:
    seed_brokers: [ "localhost:9092" ]
    topic: otel-traces
    service: order-processor

Production tracing with sampling

Configure tracing for a production environment with sampling to reduce trace volume, and with TLS and SASL authentication.

input:
  redpanda:
    seed_brokers: [ "redpanda-prod:9092" ]
    topics: [ "events" ]
    consumer_group: event-processor

pipeline:
  processors:
    - branch:
        request_map: 'root = this'
        processors:
          - mapping: 'root.enriched = true'

output:
  redpanda:
    seed_brokers: [ "redpanda-prod:9092" ]
    topic: processed_events

tracer:
  redpanda:
    seed_brokers: [ "redpanda-prod:9092" ]
    topic: otel-traces
    service: event-processor-prod
    sampling:
      enabled: true
      ratio: 0.1 # Sample 10% of traces
    tls:
      enabled: true
    sasl:
      - mechanism: SCRAM-SHA-512
        username: tracer-user
        password: ${TRACER_PASSWORD}

Multi-instance tracing

Configure tracing for multiple Redpanda Connect instances with unique service names for correlation. Each instance writes to the same topic and sets its own service name and instance tag. This configuration is for an ingestion service. A processing service uses the same configuration with service: processing-service and instance: processing-01.

tracer:
  redpanda:
    seed_brokers: [ "redpanda:9092" ]
    topic: otel-traces
    service: ingestion-service
    tags:
      environment: production
      region: us-west-2
      instance: ingestion-01

Was this helpful?

group Ask in the community

mail Share your feedback

What do you think of this page?

Let us know more:

Let us contact you about your feedback:

redpanda

Fields

acks

allow_auto_topic_creation

broker_write_max_bytes

client_id

compression

conn_idle_timeout

format

idempotent_write

max_buffered_bytes

max_buffered_records

max_in_flight_requests

max_message_bytes

metadata_max_age

partitioner

record_delivery_timeout

record_retries

request_timeout_overhead

sampling

sampling.enabled

sampling.ratio

sasl[]

sasl[].aws

sasl[].aws.credentials

sasl[].aws.credentials.from_ec2_role

sasl[].aws.credentials.id

sasl[].aws.credentials.profile

sasl[].aws.credentials.role

sasl[].aws.credentials.role_external_id

sasl[].aws.credentials.secret

sasl[].aws.credentials.token

sasl[].aws.endpoint

sasl[].aws.region

sasl[].aws.tcp

sasl[].aws.tcp.connect_timeout

sasl[].aws.tcp.keep_alive

sasl[].aws.tcp.keep_alive.count

sasl[].aws.tcp.keep_alive.idle

sasl[].aws.tcp.keep_alive.interval

sasl[].aws.tcp.tcp_user_timeout

sasl[].extensions

sasl[].mechanism

sasl[].password

sasl[].token

sasl[].username

schema_registry

schema_registry.basic_auth

schema_registry.basic_auth.enabled

schema_registry.basic_auth.password

schema_registry.basic_auth.username

schema_registry.jwt

schema_registry.jwt.claims

schema_registry.jwt.enabled

schema_registry.jwt.headers

schema_registry.jwt.private_key_file

schema_registry.jwt.signing_method

schema_registry.oauth

schema_registry.oauth.access_token

schema_registry.oauth.access_token_secret

schema_registry.oauth.consumer_key

schema_registry.oauth.consumer_secret

schema_registry.oauth.enabled

schema_registry.oauth2

schema_registry.oauth2.client_key

schema_registry.oauth2.client_secret

schema_registry.oauth2.enabled

schema_registry.oauth2.endpoint_params

schema_registry.oauth2.scopes[]

schema_registry.oauth2.token_url

schema_registry.tls

schema_registry.tls.client_certs[]

schema_registry.tls.client_certs[].cert

schema_registry.tls.client_certs[].cert_file

schema_registry.tls.client_certs[].key

schema_registry.tls.client_certs[].key_file

schema_registry.tls.client_certs[].password

`acks`

`allow_auto_topic_creation`

`broker_write_max_bytes`

`client_id`

`compression`

`conn_idle_timeout`

`format`

`idempotent_write`

`max_buffered_bytes`

`max_buffered_records`

`max_in_flight_requests`

`max_message_bytes`

`metadata_max_age`

`partitioner`

`record_delivery_timeout`

`record_retries`

`request_timeout_overhead`

`sampling`

`sampling.enabled`

`sampling.ratio`

`sasl[]`

`sasl[].aws`

`sasl[].aws.credentials`

`sasl[].aws.credentials.from_ec2_role`

`sasl[].aws.credentials.id`

`sasl[].aws.credentials.profile`

`sasl[].aws.credentials.role`

`sasl[].aws.credentials.role_external_id`

`sasl[].aws.credentials.secret`

`sasl[].aws.credentials.token`

`sasl[].aws.endpoint`

`sasl[].aws.region`

`sasl[].aws.tcp`

`sasl[].aws.tcp.connect_timeout`

`sasl[].aws.tcp.keep_alive`

`sasl[].aws.tcp.keep_alive.count`

`sasl[].aws.tcp.keep_alive.idle`

`sasl[].aws.tcp.keep_alive.interval`

`sasl[].aws.tcp.tcp_user_timeout`

`sasl[].extensions`

`sasl[].mechanism`

`sasl[].password`

`sasl[].token`

`sasl[].username`

`schema_registry`

`schema_registry.basic_auth`

`schema_registry.basic_auth.enabled`

`schema_registry.basic_auth.password`

`schema_registry.basic_auth.username`

`schema_registry.jwt`

`schema_registry.jwt.claims`

`schema_registry.jwt.enabled`

`schema_registry.jwt.headers`

`schema_registry.jwt.private_key_file`

`schema_registry.jwt.signing_method`

`schema_registry.oauth`

`schema_registry.oauth.access_token`

`schema_registry.oauth.access_token_secret`

`schema_registry.oauth.consumer_key`

`schema_registry.oauth.consumer_secret`

`schema_registry.oauth.enabled`

`schema_registry.oauth2`

`schema_registry.oauth2.client_key`

`schema_registry.oauth2.client_secret`

`schema_registry.oauth2.enabled`

`schema_registry.oauth2.endpoint_params`

`schema_registry.oauth2.scopes[]`

`schema_registry.oauth2.token_url`

`schema_registry.tls`

`schema_registry.tls.client_certs[]`

`schema_registry.tls.client_certs[].cert`

`schema_registry.tls.client_certs[].cert_file`

`schema_registry.tls.client_certs[].key`

`schema_registry.tls.client_certs[].key_file`

`schema_registry.tls.client_certs[].password`

`schema_registry.tls.enable_renegotiation`

`schema_registry.tls.root_cas`

`schema_registry.tls.root_cas_file`

`schema_registry.tls.skip_cert_verify`

`schema_registry.url`