Cloud
Also in Self-Managed

redpanda_migrator

Migrates topics, schemas, and consumer groups between Kafka and Redpanda clusters.

Pair this output with a redpanda_migrator input in the same pipeline.

The following shows all available configuration fields and their defaults.

  • Common

  • Advanced

output:
  label: ""
  redpanda_migrator:
    seed_brokers: [] # No default (required)
    schema_registry:
      url: "" # No default (required)
      timeout: 5s
      enabled: true
      interval: 5m
      include: [] # No default (optional)
      exclude: [] # No default (optional)
      subject: "" # No default (optional)
      versions: all
      include_deleted: false
      translate_ids: false
      normalize: false
      strict: false
      max_parallel_http_requests: 10
    consumer_groups:
      enabled: true
      interval: 1m
      fetch_timeout: 10s
      include: [] # No default (optional)
      exclude: [] # No default (optional)
      only_empty: false
    topic: ${! @kafka_topic }
    topic_replication_factor: 0 # No default (optional)
    sync_topic_acls: false
    headers: {} # No default (optional)
    max_in_flight: 10
output:
  label: ""
  redpanda_migrator:
    seed_brokers: [] # No default (required)
    client_id: redpanda-connect
    tls:
      enabled: false
      skip_cert_verify: false
      enable_renegotiation: false
      root_cas: ""
      root_cas_file: ""
      client_certs: []
    sasl: [] # No default (optional)
    metadata_max_age: 1m
    request_timeout_overhead: 10s
    conn_idle_timeout: 20s
    tcp:
      connect_timeout: 0s
      keep_alive:
        idle: 15s
        interval: 15s
        count: 9
      tcp_user_timeout: 0s
    partitioner: "" # No default (optional)
    idempotent_write: true
    acks: all
    compression: "" # No default (optional)
    allow_auto_topic_creation: true
    timeout: 10s
    max_message_bytes: 1MiB
    broker_write_max_bytes: 100MiB
    max_buffered_records: 10000
    max_buffered_bytes: "0"
    max_in_flight_requests: 1
    record_retries: 0
    record_delivery_timeout: 0s
    schema_registry:
      url: "" # No default (required)
      timeout: 5s
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      oauth:
        enabled: false
        consumer_key: ""
        consumer_secret: ""
        access_token: ""
        access_token_secret: ""
      basic_auth:
        enabled: false
        username: ""
        password: ""
      jwt:
        enabled: false
        private_key_file: ""
        signing_method: ""
        claims: {}
        headers: {}
      enabled: true
      interval: 5m
      include: [] # No default (optional)
      exclude: [] # No default (optional)
      subject: "" # No default (optional)
      versions: all
      include_deleted: false
      translate_ids: false
      normalize: false
      strict: false
      max_parallel_http_requests: 10
    consumer_groups:
      enabled: true
      interval: 1m
      fetch_timeout: 10s
      include: [] # No default (optional)
      exclude: [] # No default (optional)
      only_empty: false
    topic: ${! @kafka_topic }
    topic_replication_factor: 0 # No default (optional)
    sync_topic_interval: 5m
    sync_topic_acls: false
    serverless: false
    headers: {} # No default (optional)
    provenance_header: redpanda-migrator-provenance
    offset_header: redpanda-migrator-offset
    max_in_flight: 10

Requirements

When the destination cluster enforces ACLs, the destination principal needs permission to create topics and add partitions, not only to produce records. Grant these ACLs at minimum:

  • Topic CREATE, WRITE, ALTER, and DESCRIBE_CONFIGS. Cluster CREATE also authorizes topic creation.

  • When consumer group migration is enabled, consumer group READ.

  • When sync_topic_acls is enabled, cluster ALTER.

For the source-principal ACLs and full details, see Required permissions.

Multiple migrator pairs

Each migrator pair requires a unique label. Set the same label value on both the input and output within a pair. Labels must match exactly; mismatched labels prevent the input and output from coordinating.

Performance tuning

For high-throughput workloads, adjust the following settings:

On this output:

  • max_in_flight: Set to the total number of partitions being migrated. Higher values provide no benefit beyond the partition count.

On the paired redpanda_migrator input:

  • partition_buffer_bytes: Set to 2MB to increase the per-partition buffer.

  • max_yield_batch_bytes: Set to 1MB to yield larger batches.

Synchronization details

Topics

  • Topic names carry over from the source by default. Use the topic field with interpolation to rename topics at the destination.

  • The migrator creates each topic at the destination with the same partition count as the source.

  • Replication factor defaults to the source value. Set topic_replication_factor to override it.

  • The migrator copies a serverless-aware subset of topic configuration keys.

  • ACL replication is optional (sync_topic_acls):

  • The migrator does not copy ALLOW WRITE ACL entries.

  • The migrator downgrades ALLOW ALL ACL entries to ALLOW READ.

  • Resource pattern type and host filters carry over.

Schema Registry

  • Syncs once at startup, then periodically based on schema_registry.interval (default: every 5 minutes). Set schema_registry.interval: 0s for a one-time sync only.

  • Use include and exclude regex patterns to filter which subjects are migrated.

  • Use the subject field with interpolation to rename subjects at the destination.

  • Schema versions: latest (most recent version only) or all (full history). Defaults to all.

  • Include soft-deleted subjects with schema_registry.include_deleted.

  • The migrator can translate schema IDs to new destination IDs or preserve them as-is. See schema_registry.translate_ids.

  • Schema normalization is optional (schema_registry.normalize).

  • Compatibility settings carry over per-subject. Schema metadata and rules do not carry over in Serverless mode.

Consumer groups

  • Sync periodically based on consumer_groups.interval (default: every 1 minute).

  • Use include and exclude regex patterns to filter which groups are migrated.

  • By default, all groups except those in Dead state migrate. Set consumer_groups.only_empty: true to migrate only Empty state groups.

  • The migrator translates consumer offsets using timestamps. This is approximate and may be imprecise when multiple records share the same timestamp.

  • Offsets only move forward and never rewind at the destination.

  • Source and destination must have matching partition counts for each migrated group.

How it works

Each migration component synchronizes on a different schedule:

  • Topics: Sync from source on startup and every 5 minutes by default, including source topics that have no current data (for example, after retention cleanup). Configure the sync interval with sync_topic_interval, or set it to 0s to disable periodic sync. When periodic sync is disabled, topics are still created on demand when the first message arrives.

  • Schema Registry: Syncs at startup, then periodically as configured.

  • Consumer groups: Sync in the background, filtered to the topics being migrated.

Guarantees

The migrator upholds the following guarantees:

  • Creates each destination topic with the intended partition count and replication factor.

  • Never overwrites existing destination topics and logs any partition count mismatches.

  • Consumer group offsets never rewind.

  • ACL replication never grants write access at the destination.

Limitations

  • The destination cluster’s Schema Registry must be in READWRITE or IMPORT mode.

  • Offset translation is best-effort.

  • Consumer group migration requires identical partition counts at source and destination.

Metrics

Metric Name Type Labels Description

Topic migration

redpanda_migrator_topics_created_total

counter

Total topics created on destination

redpanda_migrator_topic_create_errors_total

counter

Topic creation errors

redpanda_migrator_topic_create_latency_ns

timer

Topic creation latency (ns)

Schema Registry migration

redpanda_migrator_sr_schemas_created_total

counter

Schemas created in destination registry

redpanda_migrator_sr_schema_create_errors_total

counter

Schema creation errors

redpanda_migrator_sr_schema_create_latency_ns

timer

Schema creation latency (ns)

redpanda_migrator_sr_compatibility_updates_total

counter

Compatibility level updates applied

redpanda_migrator_sr_compatibility_update_errors_total

counter

Compatibility update errors

redpanda_migrator_sr_compatibility_update_latency_ns

timer

Compatibility update latency (ns)

Consumer group migration

redpanda_migrator_cg_offsets_translated_total

counter

group

Offsets translated per consumer group

redpanda_migrator_cg_offset_translation_errors_total

counter

group

Offset translation errors per group

redpanda_migrator_cg_offset_translation_latency_ns

timer

group

Offset translation latency per group (ns)

redpanda_migrator_cg_offsets_committed_total

counter

group

Offsets committed per consumer group

redpanda_migrator_cg_offset_commit_errors_total

counter

group

Offset commit errors per group

redpanda_migrator_cg_offset_commit_latency_ns

timer

group

Offset commit latency per group (ns)

Consumer lag

redpanda_lag

gauge

topic, partition

Current consumer lag in messages for each topic partition. Shows difference between high water mark and current consumer position.

Examples

Basic migration

Migrate topics, schemas and consumer groups from source to destination.

input:
  redpanda_migrator:
    seed_brokers: ["source:9092"]
    topics: ["orders", "payments"]
    consumer_group: "migration"

output:
  redpanda_migrator:
    seed_brokers: ["destination:9092"]
    # Write to the same topic name
    topic: ${! metadata("kafka_topic") }
    schema_registry:
      url: "http://dest-registry:8081"
      translate_ids: true
    consumer_groups:
      interval: 1m

Migration to Redpanda Serverless

Migrate from Confluent/Kafka to Redpanda Cloud serverless cluster with authentication.

input:
  redpanda_migrator:
    seed_brokers: ["source-kafka:9092"]
    regexp_topics_include:
      - '.'
    regexp_topics_exclude:
      - '^_'
    consumer_group: "migrator_cg"
    schema_registry:
      url: "http://source-registry:8081"

output:
  redpanda_migrator:
    seed_brokers: ["serverless-cluster.redpanda.com:9092"]
    tls:
      enabled: true
    sasl:
      - mechanism: SCRAM-SHA-256
        username: "migrator"
        password: "migrator"
    schema_registry:
      url: "https://serverless-cluster.redpanda.com:8081"
      basic_auth:
        enabled: true
        username: "migrator"
        password: "migrator"
      translate_ids: true
    consumer_groups:
      exclude:
        - "migrator_cg"  # Exclude the migration consumer group itself
    serverless: true  # Enable serverless mode for restricted configurations

Fields

acks

The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When idempotent_write is enabled this must be set to all.

Type: string

Default: all

Option Summary

all

Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled.

leader

Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication.

none

Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost.

allow_auto_topic_creation

Enables topics to be auto created if they do not exist when fetching their metadata. If set to false, the topic must already exist.

Type: bool

Default: true

broker_write_max_bytes

The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s socket.request.max.bytes.

Type: string

Default: 100MiB

# Examples:
broker_write_max_bytes: 128MB

# ---

broker_write_max_bytes: 50mib

client_id

An identifier for the client connection. This identifier appears in broker logs and metrics, which helps you identify the Redpanda Connect instance that is connecting.

Type: string

Default: redpanda-connect

compression

Set an explicit compression type (optional). By default, the client uses snappy when the broker supports it, and falls back to none if not.

Type: string

Options: lz4, snappy, gzip, none, zstd

conn_idle_timeout

The approximate amount of time that connections can remain idle before they are closed. In the worst case, a connection can stay idle for up to twice this value. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: 20s

consumer_groups

Type: object

consumer_groups.enabled

Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed.

Type: bool

Default: true

consumer_groups.exclude[]

Regular expressions for consumer groups to exclude from offset migration. Takes precedence over include patterns. Useful for excluding system or temporary groups.

Type: array<string>

# Examples:
exclude:
  - .*-test
  - .*-temp
  - connect-.*

# ---

exclude:
  - dev-.*
  - local-.*

consumer_groups.fetch_timeout

Maximum time to wait for data when fetching records for timestamp-based offset translation. Increase for clusters with low message throughput.

Type: string

Default: 10s

# Examples:
fetch_timeout: "1s     # Fast clusters"

# ---

fetch_timeout: "10s    # Slower clusters"

consumer_groups.include[]

Regular expressions for consumer groups to include in offset migration. If empty, all groups are included (unless excluded).

Type: array<string>

# Examples:
include:
  - prod-.*
  - staging-.*

# ---

include:
  - app-.*
  - service-.*

consumer_groups.interval

How often to synchronise consumer group offsets. Regular syncing helps maintain offset accuracy during ongoing migration.

Type: string

Default: 1m

# Examples:
interval: "0s     # Disabled"

# ---

interval: "30s    # Sync every 30 seconds"

# ---

interval: "5m     # Sync every 5 minutes"

consumer_groups.only_empty

Whether to only migrate Empty consumer groups. When false (default), all statuses except Dead are included; when true, only Empty groups are migrated.

Type: bool

Default: false

headers

Custom headers to add to migrated records, keyed by header name with interpolated string values. Useful for injecting metadata such as processing timestamps or latency measurements that should surface as header values on the destination cluster. A custom header name that collides with provenance_header or offset_header is ignored, so those migration-critical headers are always protected.

This field supports interpolation functions.

Type: object<string>

# Examples:
headers:
  x-migration-latency-ms: ${! timestamp_unix_milli() - meta("kafka_timestamp_ms") }
  x-migration-processed-at: ${! timestamp_unix_milli() }

idempotent_write

Enable the idempotent write producer option. When enabled, the producer initializes a producer ID and uses it to guarantee exactly-once semantics per partition, so retries do not produce duplicates. This option requires the IDEMPOTENT_WRITE permission on the CLUSTER resource. Disable this option if the IDEMPOTENT_WRITE permission is unavailable, for example in some managed Kafka services or Redpanda clusters with strict ACLs. Disabling this option only affects retry behavior: duplicates may occur on producer retries, but the pipeline continues to function normally.

Type: bool

Default: true

max_buffered_bytes

The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered. Set to 0 to disable the byte-level limit (only max_buffered_records applies). This limit is checked after max_buffered_records.

Type: string

Default: 0

# Examples:
max_buffered_bytes: 256MB

# ---

max_buffered_bytes: 50mib

max_buffered_records

The maximum number of records the client will buffer in memory before blocking. When this limit is reached, Produce() calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls.

Type: int

Default: 10000

max_in_flight

The maximum number of batches to send in parallel at any given time. Increase this value to improve throughput during migration.

For optimal performance, set this to the total number of partitions being migrated in parallel (up to all partitions in the cluster). Setting it higher than the number of consumed partitions provides no additional benefit.

For example, if you are migrating 100 partitions, set max_in_flight: 100 for maximum throughput.

Type: int

Default: 10

# Examples:
max_in_flight: "64  # For a cluster with 64 partitions"

# ---

max_in_flight: "128 # For multiple topics with combined 128 partitions"

max_in_flight_requests

The maximum number of produce requests in flight per broker connection. While idempotent_write is enabled (the default) this must be 1, as the client relies on a single in-flight request per broker to guarantee ordering. To use higher values you must set idempotent_write to false, which allows requests to be pipelined for throughput but may cause duplicate and out-of-order delivery on retries. Note that this is distinct from the output’s max_in_flight field, which counts message batches being written in parallel rather than produce requests on the wire. A value of 1 is not a throughput ceiling: records from concurrent writes are coalesced into fewer, larger produce requests.

Type: int

Default: 1

max_message_bytes

The maximum size of a produced record batch in bytes. A MESSAGE_TOO_LARGE error is returned if a batch exceeds this limit. This field maps to the max.message.bytes Kafka property. Ensure the Redpanda broker’s kafka_batch_max_bytes property is at least as large as this value, see kafka_batch_max_bytes.

Type: string

Default: 1MiB

# Examples:
max_message_bytes: 100MB

# ---

max_message_bytes: 50mib

metadata_max_age

The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes.

This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics.

Type: string

Default: 1m

offset_header

The name of a message header to add to migrated records. This header contains the source offset, enabling exact consumer group offset translation during migration.

If this field is empty, no offset header is added and exact offset translation is disabled. Consumer groups are still migrated using timestamp-based positioning, which works well for most cases but may be imprecise for consumer groups in the Empty state when multiple records share the same timestamp (timestamps have millisecond resolution).

Exact offset translation only applies to consumer groups in the Empty state, which have no active members. Other groups always use timestamp-based positioning. The default value enables exact offset translation.

This header is only added when consumer group migration is enabled.

Type: string

Default: redpanda-migrator-offset

partitioner

Override the default murmur2 hashing partitioner.

Type: string

Option Summary

least_backup

Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch.

manual

Manually select a partition for each message, requires the field partition to be specified.

murmur2_hash

Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on.

round_robin

Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions.

provenance_header

Header name to add to migrated records indicating their source cluster. When set, each migrated record that does not already carry this header receives it, with the source cluster’s ID (from the cluster metadata) as the value. Downstream systems can use the header to track record origins. A record that already carries the header keeps its existing value. A record whose header value is the destination cluster’s ID is skipped, so it is not sent back to the cluster it came from. A record whose header value is empty or equal to the source cluster’s ID causes an error. If empty, no provenance header is added and no provenance checks run.

Type: string

Default: redpanda-migrator-provenance

record_delivery_timeout

The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s delivery.timeout.ms. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to 0s for no timeout (the default). With idempotent_write enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers.

Type: string

Default: 0s

record_retries

The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to 0 for unlimited retries (the default). With idempotent_write enabled, retries are only enforced when safe to do so without creating invalid sequence numbers.

Type: int

Default: 0

request_timeout_overhead

Additional time to apply as overhead when calculating request deadlines. For most requests, the deadline is this overhead alone. For requests that define their own timeout field, the overhead is added on top of that timeout, which helps prevent premature timeouts.

This field is roughly equivalent to Apache Kafka’s request.timeout.ms parameter, but grants extra time to requests that have timeout fields.

Type: string

Default: 10s

sasl[]

Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail.

Type: array<object>

# Examples:
sasl:
  - mechanism: SCRAM-SHA-512
    password: bar
    username: foo

sasl[].aws

Contains AWS-specific fields for when sasl.mechanism is set to AWS_MSK_IAM.

Type: object

sasl[].aws.credentials

Manually configure the AWS credentials to use (optional). For more information, see the Amazon Web Services guide.

Type: object

sasl[].aws.credentials.from_ec2_role

Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance.

Type: bool

sasl[].aws.credentials.id

The ID of the AWS credentials to use.

Type: string

sasl[].aws.credentials.profile

The profile from ~/.aws/credentials to use.

Type: string

sasl[].aws.credentials.role

The ARN of the role to assume.

Type: string

sasl[].aws.credentials.role_external_id

An external ID to use when assuming a role.

Type: string

sasl[].aws.credentials.secret

The secret for the AWS credentials in use.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

sasl[].aws.credentials.token

The token for the AWS credentials in use. Required only when using short-term credentials.

Type: string

sasl[].aws.endpoint

A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints.

Type: string

sasl[].aws.region

The AWS region in which your resources are hosted.

Type: string

sasl[].aws.tcp

Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:

  • Unresponsive hosts: Set connect_timeout to limit how long a connection attempt can take (the default 0s sets no limit)

  • Long-lived connections: Configure keep_alive settings to detect and recover from stale connections

  • Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives

  • Linux systems with specific requirements: Use tcp_user_timeout (Linux 2.6.37+) to control data acknowledgment timeouts

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Type: object

sasl[].aws.tcp.connect_timeout

Maximum amount of time a dial will wait for a connect to complete. Zero disables.

Type: string

Default: 0s

sasl[].aws.tcp.keep_alive

TCP keep-alive probe configuration.

Type: object

sasl[].aws.tcp.keep_alive.count

Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.

Type: int

Default: 9

sasl[].aws.tcp.keep_alive.idle

Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.

Type: string

Default: 15s

sasl[].aws.tcp.keep_alive.interval

Duration between keep-alive probes. Zero defaults to 15s.

Type: string

Default: 15s

sasl[].aws.tcp.tcp_user_timeout

Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep_alive.idle must be greater than this value per RFC 5482. Zero disables.

Type: string

Default: 0s

sasl[].extensions

Key/value pairs to add to OAUTHBEARER authentication requests.

Type: object<string>

sasl[].mechanism

The SASL mechanism to use for authentication.

Type: string

Option Summary

AWS_MSK_IAM

AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library.

OAUTHBEARER

OAuth Bearer based authentication.

PLAIN

Plain text authentication.

REDPANDA_CLOUD_SERVICE_ACCOUNT

Redpanda Cloud Service Account authentication when running in Redpanda Cloud.

SCRAM-SHA-256

SCRAM based authentication as specified in RFC5802.

SCRAM-SHA-512

SCRAM based authentication as specified in RFC5802.

none

Disable sasl authentication

sasl[].password

The password to use for PLAIN or SCRAM-* authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

sasl[].token

The token to use for a single session’s OAUTHBEARER authentication.

Type: string

Default: ""

sasl[].username

The username to use for PLAIN or SCRAM-* authentication.

Type: string

Default: ""

schema_registry

Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters.

Type: object

schema_registry.basic_auth

Configure basic authentication for requests from this component.

Type: object

schema_registry.basic_auth.enabled

Whether to use basic authentication in requests.

Type: bool

Default: false

schema_registry.basic_auth.password

The password to use for authentication. Used together with username for basic authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

schema_registry.basic_auth.username

The username of the account credentials to authenticate as. Used together with password for basic authentication.

Type: string

Default: ""

schema_registry.enabled

Whether schema registry migration is enabled. When disabled, no schema operations are performed.

Type: bool

Default: true

schema_registry.exclude[]

Regular expressions for schema subjects to exclude from migration. Takes precedence over include patterns. Note: the migrator consumer group is always ignored.

Type: array<string>

# Examples:
exclude:
  - .*-test
  - .*-temp

# ---

exclude:
  - dev-.*
  - local-.*

schema_registry.include[]

Regular expressions for schema subjects to include in migration. If empty, all subjects are included (unless excluded). Note: the migrator consumer group is always ignored.

Type: array<string>

# Examples:
include:
  - prod-.*
  - staging-.*

# ---

include:
  - user-.*
  - order-.*

schema_registry.include_deleted

Whether to include soft-deleted schemas in migration. Useful for complete migration but may not be supported by all schema registries.

Type: bool

Default: false

schema_registry.interval

How often to synchronise schema registry subjects. Set to 0s for one-time sync at startup only.

Type: string

Default: 5m

# Examples:
interval: "0s     # One-time sync only"

# ---

interval: "5m     # Sync every 5 minutes"

# ---

interval: "30m    # Sync every 30 minutes"

schema_registry.jwt

Beta

Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWTs provide secure, stateless authentication between services.

Type: object

schema_registry.jwt.claims

A map of claims to include in the JWT. Claims pass the identity of the authenticated entity to the service provider.

Type: object

Default: {}

schema_registry.jwt.enabled

Whether to use JWT authentication in requests.

Type: bool

Default: false

schema_registry.jwt.headers

Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing.

Type: object

Default: {}

schema_registry.jwt.private_key_file

Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the signing_method field.

Type: string

Default: ""

schema_registry.jwt.signing_method

The cryptographic algorithm used to sign the JWT. Supported algorithms are RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the private_key_file field.

Type: string

Default: ""

schema_registry.max_parallel_http_requests

Maximum number of parallel HTTP requests to the schema registry. Controls concurrency when syncing multiple schemas.

Type: int

Default: 10

schema_registry.normalize

Whether to normalize schemas when creating them in the destination registry.

Type: bool

Default: false

schema_registry.oauth

Configure OAuth version 1.0 authentication for secure API access.

Type: object

schema_registry.oauth.access_token

The value used to gain access to the protected resources on behalf of the user.

Type: string

Default: ""

schema_registry.oauth.access_token_secret

The secret that establishes ownership of the access_token in OAuth 1.0 authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

schema_registry.oauth.consumer_key

The value used to identify this component or client to the service provider.

Type: string

Default: ""

schema_registry.oauth.consumer_secret

The secret that establishes ownership of the consumer key in OAuth 1.0 authentication.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

schema_registry.oauth.enabled

Whether to enable OAuth version 1.0 authentication for requests.

Type: bool

Default: false

schema_registry.strict

Error on unknown schema IDs. Only relevant when translate_ids is true. When false (default), unknown schema IDs are passed through unchanged, allowing migration of topics with mixed message formats. Note: messages with 0-byte prefixes (for example, protobuf) cannot be distinguished from schema registry headers and may fail when strict is enabled.

Type: bool

Default: false

schema_registry.subject

Template for transforming subject names during migration. Use interpolation to rename subjects systematically.

This field supports interpolation functions.

Type: string

# Examples:
subject: prod_${! metadata("schema_registry_subject") }

# ---

subject: ${! metadata("schema_registry_subject") | replace("dev_", "prod_") }

schema_registry.timeout

HTTP client timeout for schema registry requests.

Type: string

Default: 5s

schema_registry.tls

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

schema_registry.tls.client_certs[]

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

  • Inline PEM data using both cert and key or

  • File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

schema_registry.tls.client_certs[].cert

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

schema_registry.tls.client_certs[].cert_file

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

schema_registry.tls.client_certs[].key

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

schema_registry.tls.client_certs[].key_file

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

schema_registry.tls.client_certs[].password

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

schema_registry.tls.enable_renegotiation

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

schema_registry.tls.enabled

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

schema_registry.tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

schema_registry.tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

schema_registry.tls.skip_cert_verify

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

schema_registry.translate_ids

Whether to translate schema IDs during migration.

Type: bool

Default: false

schema_registry.url

The base URL of the schema registry service. Required for schema migration functionality.

Type: string

# Examples:
url: http://localhost:8081

# ---

url: https://schema-registry.example.com:8081

schema_registry.versions

Which schema versions to migrate. 'latest' migrates only the current version, 'all' migrates complete version history for better compatibility.

Type: string

Default: all

Options: latest, all

seed_brokers[]

A list of broker addresses used to establish connections. If an item of the list contains commas, it is expanded into multiple addresses.

Type: array<string>

# Examples:
seed_brokers:
  - "localhost:9092"

# ---

seed_brokers:
  - "foo:9092"
  - "bar:9092"

# ---

seed_brokers:
  - "foo:9092,bar:9092"

serverless

Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by serverless environments.

Type: bool

Default: false

sync_topic_acls

Whether to synchronise topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts.

Type: bool

Default: false

sync_topic_interval

How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow. Set to 0s to disable periodic sync (topics are still created on first message).

Type: string

Default: 5m

# Examples:
sync_topic_interval: "0s     # Disable periodic sync"

# ---

sync_topic_interval: "1m     # Sync every minute"

# ---

sync_topic_interval: "5m     # Sync every 5 minutes"

tcp

Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for:

  • Unresponsive hosts: Set connect_timeout to limit how long a connection attempt can take (the default 0s sets no limit)

  • Long-lived connections: Configure keep_alive settings to detect and recover from stale connections

  • Unstable networks: Tune keep-alive probes to balance between quick failure detection and avoiding false positives

  • Linux systems with specific requirements: Use tcp_user_timeout (Linux 2.6.37+) to control data acknowledgment timeouts

Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements.

Type: object

tcp.connect_timeout

Maximum amount of time a dial will wait for a connect to complete. Zero disables.

Type: string

Default: 0s

tcp.keep_alive

TCP keep-alive probe configuration.

Type: object

tcp.keep_alive.count

Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9.

Type: int

Default: 9

tcp.keep_alive.idle

Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes.

Type: string

Default: 15s

tcp.keep_alive.interval

Duration between keep-alive probes. Zero defaults to 15s.

Type: string

Default: 15s

tcp.tcp_user_timeout

Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep_alive.idle must be greater than this value per RFC 5482. Zero disables.

Type: string

Default: 0s

timeout

The maximum period of time to wait for message sends before abandoning the request and retrying.

Type: string

Default: 10s

tls

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

tls.client_certs[]

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

  • Inline PEM data using both cert and key or

  • File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: array<object>

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

tls.client_certs[].cert

The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the key field when using inline PEM data for mTLS client certificates.

Type: string

Default: ""

tls.client_certs[].cert_file

The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the key_file field when using file-based configuration for mTLS client certificates.

Type: string

Default: ""

tls.client_certs[].key

Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the cert field. Use this field together with cert when providing certificate data inline rather than through files.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

tls.client_certs[].key_file

Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the cert_file field. Use this field together with cert_file when loading certificate data from files.

Type: string

Default: ""

tls.client_certs[].password

The password to use for the private key (specified in the key or key_file fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management.

The pbeWithMD5AndDES-CBC algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

tls.enable_renegotiation

Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

tls.enabled

Whether to enable TLS for secure connections. Set to true to enable TLS encryption. Required to be true for other TLS options (like client_certs, root_cas, etc.) to take effect.

Type: bool

Default: false

tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

tls.skip_cert_verify

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

topic

The topic to write messages to. To derive destination topic names from source topics, use interpolation. The source topic name is available in the kafka_topic metadata field.

This field supports interpolation functions.

Type: string

Default: ${! @kafka_topic }

# Examples:
topic: prod_${! @kafka_topic }

topic_replication_factor

The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes.

Type: int

# Examples:
topic_replication_factor: "3"

# ---

topic_replication_factor: "1  # For single-node clusters"