Cloud
Also in Self-Managed

cyborgdb

Inserts items into a CyborgDB encrypted vector index.

This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization.

All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure.

  • Common

  • Advanced

output:
  label: ""
  cyborgdb:
    max_in_flight: 64
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
    host: "" # No default (required)
    api_key: "" # No default (required)
    index_name: redpanda-vectors
    index_key: "" # No default (required)
    operation: upsert
    id: "" # No default (required)
    vector_mapping: "" # No default (optional)
    metadata_mapping: "" # No default (optional)
output:
  label: ""
  cyborgdb:
    max_in_flight: 64
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
      processors: [] # No default (optional)
    host: "" # No default (required)
    api_key: "" # No default (required)
    index_name: redpanda-vectors
    index_key: "" # No default (required)
    create_if_missing: false
    operation: upsert
    id: "" # No default (required)
    vector_mapping: "" # No default (optional)
    metadata_mapping: "" # No default (optional)

Fields

api_key

The API key for authenticating with the CyborgDB service. This key identifies your account and provides access to your CyborgDB indexes. Keep this key secure and avoid exposing it in logs or version control.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

batching

Configure a batching policy.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

batching.byte_size

The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).

Set to 0 to disable size-based batching. When disabled, messages are flushed based on other conditions (such as count or period).

Type: int

Default: 0

batching.check

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

batching.count

The number of messages at which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

batching.period

The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s. Supported time units are ns, us, ms, s, m, and h.

Type: string

Default: ""

# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms

batching.processors[]

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.

Type: array<processor>

# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array

create_if_missing

Whether to create the index if it doesn’t exist. When enabled, CyborgDB automatically detects the vector dimensions from your data and optimizes the index configuration for performance. This is useful for development and testing environments.

Type: bool

Default: false

host

The host URL for the CyborgDB instance, including the port number if required. The scheme is optional: if the value does not start with http:// or https://, this output uses https://.

Type: string

# Examples:
host: api.cyborg.com

# ---

host: localhost:8000

id

The unique identifier for each vector entry, for example ${! json("id") }. This ID is used to update existing vectors during upsert operations or to specify which vectors to delete.

This field supports interpolation functions.

Type: string

index_key

The base64-encoded encryption key for the CyborgDB index. This key must be exactly 32 bytes when decoded from base64. All vector data is encrypted client-side using this key before transmission, ensuring complete data privacy. Store this key securely as it cannot be recovered if lost.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

# Examples:
index_key: your-base64-encoded-32-byte-key

index_name

The name of the CyborgDB index to write vectors to. If the index doesn’t exist and create_if_missing is enabled, CyborgDB will create it automatically with optimized settings based on your data.

Type: string

Default: redpanda-vectors

max_in_flight

The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput.

Type: int

Default: 64

metadata_mapping

An optional Bloblang mapping that extracts metadata to associate with the vector entry. The metadata can contain any JSON-serializable data that helps identify or categorize the vector. This data is stored encrypted alongside the vector.

Type: string

# Examples:
metadata_mapping: root = @

# ---

metadata_mapping: root = metadata()

# ---

metadata_mapping: 'root = {"summary": this.summary, "category": this.category}'

operation

The operation to perform against the CyborgDB index. Supported operations:

  • upsert: Insert new vectors or update existing ones (requires vector_mapping)

  • delete: Remove vectors from the index by id

Type: string

Default: upsert

Options: upsert, delete

vector_mapping

A Bloblang mapping that extracts the vector from the message. The result must be an array of floating-point numbers representing the vector embeddings. This field is required for the upsert operation.

Type: string

# Examples:
vector_mapping: root = this.embeddings_vector

# ---

vector_mapping: root = [1.2, 0.5, 0.76]