cyborgdb

Available in: Self-Managed

Inserts items into a CyborgDB encrypted vector index.

Introduced in version 4.66.0

  • Common

  • Advanced

outputs:
  label: ""
  cyborgdb:
    max_in_flight: 64
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
      processors: [] # No default (optional)
    host: "" # No default (required)
    api_key: "" # No default (required)
    index_name: redpanda-vectors
    index_key: "" # No default (required)
    operation: upsert
    id: "" # No default (required)
    vector_mapping: "" # No default (optional)
    metadata_mapping: "" # No default (optional)
outputs:
  label: ""
  cyborgdb:
    max_in_flight: 64
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
      processors: [] # No default (optional)
    host: "" # No default (required)
    api_key: "" # No default (required)
    index_name: redpanda-vectors
    index_key: "" # No default (required)
    create_if_missing: false
    operation: upsert
    id: "" # No default (required)
    vector_mapping: "" # No default (optional)
    metadata_mapping: "" # No default (optional)

This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization.

All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure.

Fields

api_key

The API key for authenticating with the CyborgDB service. This key identifies your account and provides access to your CyborgDB indexes. Keep this key secure and avoid exposing it in logs or version control.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

batching

Allows you to configure a batching policy.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s
batching:
  count: 10
  period: 1s
batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

batching.byte_size

An amount of bytes at which the batch should be flushed. If 0 disables size based batching.

Type: int

Default: 0

batching.check

A Bloblang query that should return a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

batching.count

A number of messages at which the batch should be flushed. If 0 disables count based batching.

Type: int

Default: 0

batching.period

A period in which an incomplete batch should be flushed regardless of its size.

Type: string

Default: ""

# Examples:
period: 1s
period: 1m
period: 500ms

batching.processors[]

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.

Type: processor

# Examples:
processors:
  - archive:
      format: concatenate

  - archive:
      format: lines

  - archive:
      format: json_array

create_if_missing

Whether to create the index if it doesn’t exist. When enabled, CyborgDB automatically detects the vector dimensions from your data and optimizes the index configuration for performance. This is useful for development and testing environments.

Type: bool

Default: false

host

The host URL for the CyborgDB instance. This should include the protocol (https://) and port number if required. For example: https://api.cyborgdb.com or https://localhost:8080.

Type: string

# Examples:
host: api.cyborg.com
host: localhost:8000

id

A Bloblang mapping that determines the unique identifier for each vector entry. This ID is used to update existing vectors during upsert operations or to specify which vectors to delete. If not provided, CyborgDB will generate unique IDs automatically.

This field supports interpolation functions.

Type: string

index_key

The base64-encoded encryption key for the CyborgDB index. This key must be exactly 32 bytes when decoded from base64. All vector data is encrypted client-side using this key before transmission, ensuring complete data privacy. Store this key securely as it cannot be recovered if lost.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

# Examples:
index_key: your-base64-encoded-32-byte-key

index_name

The name of the CyborgDB index to write vectors to. If the index doesn’t exist and create_if_missing is enabled, CyborgDB will create it automatically with optimized settings based on your data.

Type: string

Default: redpanda-vectors

max_in_flight

The maximum number of messages to have in flight at a given time. Increase this to improve throughput.

Type: int

Default: 64

metadata_mapping

An optional Bloblang mapping that extracts metadata to associate with the vector entry. The metadata can contain any JSON-serializable data that helps identify or categorize the vector. This data is stored encrypted alongside the vector.

Type: string

# Examples:
metadata_mapping: root = @
metadata_mapping: root = metadata()
metadata_mapping: root = {"summary": this.summary, "category": this.category}

operation

The operation to perform against the CyborgDB index. Supported operations:

  • upsert: Insert new vectors or update existing ones (requires vector_mapping)

  • delete: Remove vectors from the index (requires id)

  • query: Search for similar vectors (requires vector_mapping)

Type: string

Default: upsert

Options: upsert, delete

vector_mapping

A Bloblang mapping that extracts the vector from the message. The result must be an array of floating-point numbers representing the vector embeddings. This field is required for upsert and query operations.

Type: string

# Examples:
vector_mapping: root = this.embeddings_vector
vector_mapping: root = [1.2, 0.5, 0.76]