Connect

Templating

Templates are an experimental feature and are subject to change outside major version releases.

Templates are a way to define new Redpanda Connect components (similar to plugins) that are implemented by generating a Redpanda Connect configuration snippet from pre-defined parameter fields. This is useful when a common pattern of Redpanda Connect configuration is used but with varying parameters each time.

A template is defined in a YAML file that can be imported when Redpanda Connect runs using the flag -t:

rpk connect run -t "./templates/*.yaml" ./config.yaml

The template describes the type of the component and configuration fields that can be used to customize it, followed by a Bloblang mapping that translates an object containing those fields into a Redpanda Connect configuration structure. This allows you to use logic to generate more complex configurations:

  • Template

  • Configuration

  • Result

name: aws_sqs_list
type: input

fields:
  - name: urls
    type: string
    kind: list
  - name: region
    type: string
    default: us-east-1

mapping: |
  root.broker.inputs = this.urls.map_each(url -> {
    "aws_sqs": {
      "url": url,
      "region": this.region,
    }
  })
input:
  aws_sqs_list:
    urls:
      - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1
      - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2

pipeline:
  processors:
    - mapping: |
        root.id = uuid_v4()
        root.foo = this.inner.foo
        root.body = this.outer
input:
  broker:
    inputs:
      - aws_sqs:
          url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1
          region: us-east-1
      - aws_sqs:
          url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2
          region: us-east-1

pipeline:
  processors:
    - mapping: |
        root.id = uuid_v4()
        root.foo = this.inner.foo
        root.body = this.outer

You can see more examples of templates on GitHub.

Enum field options

When fields[].type is set to string_enum or string_annotated_enum, the fields[].options field is required:

  • For string_enum, provide a non-empty array of string values.

  • For string_annotated_enum, provide a map of value: "description" pairs. The descriptions annotate the behavior of each value in the documentation.

fields:
  - name: mode
    type: string_enum
    options: ["fast", "balanced", "safe"]

  - name: delivery
    type: string_annotated_enum
    options:
      at_least_once: "May duplicate messages but never loses them."
      exactly_once: "Stronger guarantees but with a higher cost and constraints."

Test a template

Use the tests field to define unit tests for a template. The following examples show a test template, an associated test configuration, and the commands that run the tests.

test.template.yaml
name: basictemplate
type: input

mapping: |
  root.generate = {
    "count": 1,
    "mapping": "root = \"" + @label.or("") + "\""
  }

tests:
  - name: basictemplate test
    label: quack
    config: {}
    expected:
      generate:
        count: 1
        mapping: root = "quack"
testconfig.yaml
input:
  label: meow
  basictemplate: {}

To lint the template and run its tests, then run the test configuration with the template, use rpk connect template lint and rpk connect run:

rpk connect template lint test.template.yaml
rpk connect run -t "./test.template.yaml" ./testconfig.yaml

Fields

categories[]

An optional list of tags, which are used for arbitrarily grouping components in documentation.

Type: array<string>

Default: []

description

A longer form description of the component and how to use it.

Type: string

Default: ""

fields[]

The configuration fields of the template, fields specified here will be parsed from a Redpanda Connect config and will be accessible from the template mapping.

Type: array<object>

fields[].advanced

Whether this field is considered advanced.

Type: bool

Default: false

fields[].default

An optional default value for the field. If a default value is not specified then a configuration without the field is considered incorrect.

Type: unknown

fields[].description

A description of the field.

Type: string

Default: ""

fields[].kind

The kind of the field.

Type: string

Default: scalar

Options: scalar, map, list

fields[].name

The name of the field.

Type: string

fields[].options

List of options for string_enum fields or map of annotated options for string_annotated_enum fields

Type: unknown

fields[].type

The scalar type of the field.

Type: string

Option Summary

string

standard string type

string_enum

string type which can have one of a discrete list of values

string_annotated_enum

string type which can have one of a discrete list of values, where each value must be accompanied by a description that annotates its behaviour in the documentation

int

standard integer type

float

standard float type

bool

standard boolean type

bloblang

bloblang mapping

unknown

allows for nesting arbitrary configuration inside of a field

mapping

A Bloblang mapping that translates the fields of the template into a valid Redpanda Connect configuration for the target component type.

Type: string

metrics_mapping

An optional Bloblang mapping that allows you to rename or prevent certain metrics paths from being exported. For more information check out the metrics documentation. When metric paths are created, renamed and dropped a trace log is written, enabling TRACE level logging is therefore a good way to diagnose path mappings.

Invocations of this mapping are able to reference a variable $label in order to obtain the value of the label provided to the template config. This allows you to match labels with the root of the config.

Type: string

Default: ""

# Examples:
metrics_mapping: this.replace("input", "source").replace("output", "sink")

# ---

metrics_mapping: |-
  root = if ![
    "input_received",
    "input_latency",
    "output_sent"
  ].contains(this) { deleted() }

name

The name of the component this template will create.

Type: string

status

The stability of the template describing the likelihood that the configuration spec of the template, or it’s behavior, will change.

Type: string

Default: stable

Option Summary

stable

This template is stable and will therefore not change in a breaking way outside of major version releases.

beta

This template is beta and will therefore not change in a breaking way unless a major problem is found.

experimental

This template is experimental and therefore subject to breaking changes outside of major version releases.

deprecated

This template has been deprecated and should no longer be used.

summary

A short summary of the component.

Type: string

Default: ""

tests[]

Optional unit test definitions for the template that verify certain configurations produce valid configs. These tests are executed with the command rpk connect template lint.

Type: array<object>

Default: []

tests[].config

A configuration to run this test with, the config resulting from applying the template with this config will be linted.

Type: object

tests[].expected

An optional configuration describing the expected result of applying the template, when specified the result will be diffed and any mismatching fields will be reported as a test error.

Type: object

tests[].label

A label to assign to this template when running the test.

Type: string

Default: ""

tests[].name

A name to identify the test.

Type: string

type

The type of the component this template will create.

Type: string

Options: cache, input, output, processor, rate_limit