Connect

ockam_kafka

Type:

Available in: Self-Managed

Uses Ockam to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the ockam_kafka output or by creating a Kafka Portal Inlet using Ockam Command.

Ockam Secure Channels guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages.

You can use Ockam to encrypt whole messages, or specific fields in a message.

Introduced in version 4.33.0.

  • Common

  • Advanced

inputs:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      topics: [] # No default (optional)
      regexp_topics_include: [] # No default (optional)
      regexp_topics_exclude: [] # No default (optional)
      rack_id: ""
      instance_id: ""
      rebalance_timeout: 45s
      session_timeout: 1m
      heartbeat_interval: 3s
      start_offset: earliest
      fetch_max_bytes: 50MiB
      fetch_max_wait: 5s
      fetch_min_bytes: 1B
      fetch_max_partition_bytes: 1MiB
      transaction_isolation_level: read_uncommitted
      consumer_group: "" # No default (optional)
      checkpoint_limit: 1024
      commit_period: 5s
      multi_header: false
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      topic_lag_refresh_period: 5s
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
    encrypted_fields: []
inputs:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      topics: [] # No default (optional)
      regexp_topics_include: [] # No default (optional)
      regexp_topics_exclude: [] # No default (optional)
      rack_id: ""
      instance_id: ""
      rebalance_timeout: 45s
      session_timeout: 1m
      heartbeat_interval: 3s
      start_offset: earliest
      fetch_max_bytes: 50MiB
      fetch_max_wait: 5s
      fetch_min_bytes: 1B
      fetch_max_partition_bytes: 1MiB
      transaction_isolation_level: read_uncommitted
      consumer_group: "" # No default (optional)
      checkpoint_limit: 1024
      commit_period: 5s
      multi_header: false
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      topic_lag_refresh_period: 5s
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
    encrypted_fields: []

Fields

allow

Use in conjunction with the route_to_kafka_outlet field to specify an access control policy for the Kafka Portal Outlet.

For example, setting this value to kafka_us_east forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute kafka_us_east=true.

Type: string

Default: self

allow_producer

Specify an access control policy for producers.

For example, setting this value to orders_producer forces the producer to present an Ockam credential, which confirms that the producer has the attribute orders_producer=true.

Type: string

Default: self

disable_content_encryption

Disables Kafka message encryption.

If this value is set to true:

  • Only message payloads remain unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled.

  • All other ockam_kafka inlets and outlets in a topic must also have their settings set to true.

Type: bool

Default: false

encrypted_fields[]

The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted.

Type: array

Default: []

enrollment_ticket

The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional).

You can generate a new ticket using the ockam project ticket command.

Type: string

identity_name

The name of the Ockam identity to use. If this value is not provided, the default Ockam identity is automatically generated and used (optional).

Type: string

kafka

Type: object

kafka.batching

Configure a batching policy for individual topic partitions. This allows the input to batch messages together before flushing them for processing. Batching may improve performance and is useful for windowed processing as it preserves the ordering of topic partitions.

Type: object

# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

kafka.batching.byte_size

The number of bytes at which the batch is flushed. Set to 0 to disable size-based batching.

Type: int

Default: 0

kafka.batching.check

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples:
check: this.type == "end_of_transaction"

kafka.batching.count

The number of messages after which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

kafka.batching.period

The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: ""

# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms

kafka.batching.processors[]

For aggregating and archiving message batches, you can add a list of processors to apply to a batch as it is flushed (optional). All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches.

Type: processor

# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array

kafka.checkpoint_limit

The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied.

When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the checkpoint_limit value.

Type: int

Default: 1024

kafka.commit_period

The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown.

Type: string

Default: 5s

kafka.consumer_group

Assign a consumer group for the processing of messages (optional). When this value is set:

  • Partitions of specified topics are automatically distributed across consumers sharing a consumer group.

  • Partition offsets are automatically committed and resumed under this name.

Consumer groups are not supported when explicit partitions to consume from are specified in the topics field.

Type: string

kafka.fetch_max_bytes

The maximum size of a message batch (in bytes) that a broker tries to send during a client fetch. If individual records exceed the fetch_max_bytes value, brokers will still send them.

Type: string

Default: 50MiB

kafka.fetch_max_partition_bytes

The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting fetch.max.partition.bytes.

If a single batch is larger than the fetch_max_partition_bytes value, the batch is still sent so that the client can make progress.

Type: string

Default: 1MiB

kafka.fetch_max_wait

The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (fetch_min_bytes).

Type: string

Default: 5s

kafka.fetch_min_bytes

The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting fetch.min.bytes.

Type: string

Default: 1B

kafka.heartbeat_interval

When you specify a consumer_group, heartbeat_interval sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active.

You must set heartbeat_interval to less than one-third of session_timeout.

This field is equivalent to the Java heartbeat.interval.ms setting and accepts Go duration format strings such as 10s or 2m.

Type: string

Default: 3s

kafka.instance_id

When you specify a consumer_group, assign a unique value to instance_id to define the group’s static membership, which can prevent unnecessary rebalances during reconnections.

When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID.

Type: string

Default: ""

kafka.multi_header

Decode headers into lists to allow the handling of multiple values with the same key.

Type: bool

Default: false

kafka.rack_id

A rack identifier for this client.

Type: string

Default: ""

kafka.rebalance_timeout

When you specify a consumer_group, rebalance_timeout sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: 45s

kafka.regexp_topics_exclude[]

A list of regular expression patterns for excluding topics when regex mode is enabled (via regexp_topics or regexp_topics_include). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns.

Type: array

kafka.regexp_topics_include[]

A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the metadata_max_age interval. This enables regex mode and cannot be used together with the topics field. Use regexp_topics_exclude to exclude specific patterns.

Type: array

# Examples:
regexp_topics_include:
  - logs_.*
  - metrics_.*

# ---

regexp_topics_include:
  - "events_[0-9]+"

kafka.seed_brokers[]

A list of broker addresses to connect to (optional). List items that contain commas are expanded into multiple addresses.

Type: array

# Examples:
seed_brokers:
  - "localhost:9092"

# ---

seed_brokers:
  - "foo:9092"
  - "bar:9092"

# ---

seed_brokers:
  - "foo:9092,bar:9092"

kafka.session_timeout

When you specify a consumer_group, session_timeout sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: 1m

kafka.start_offset

Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the OffsetOutOfRange error is seen during a fetch.

Type: string

Default: earliest

Option Summary

committed

Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option

earliest

Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option.

latest

Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option.

kafka.tls

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include enabled to enable TLS, client_certs for mTLS authentication, root_cas/root_cas_file for custom certificate authorities, and skip_cert_verify for development environments.

Type: object

kafka.tls.client_certs[]

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set tls.enabled: true for the client certificates to take effect.

Certificate pairing rules: For each certificate item, provide either:

  • Inline PEM data using both cert and key or

  • File paths using both cert_file and key_file.

Mixing inline and file-based values within the same item is not supported.

Type: object

Default: []

# Examples:
client_certs:
  - cert: foo
    key: bar


# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

kafka.tls.client_certs[].cert

A plain text certificate to use.

Type: string

Default: ""

kafka.tls.client_certs[].cert_file

The path of a certificate to use.

Type: string

Default: ""

kafka.tls.client_certs[].key

A plain text certificate key to use.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

kafka.tls.client_certs[].key_file

The path of a certificate key to use.

Type: string

Default: ""

kafka.tls.client_certs[].password

A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete pbeWithMD5AndDES-CBC algorithm is not supported for the PKCS#8 format.

Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}

kafka.tls.enable_renegotiation

Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Requires version 3.45.0 or later.

Type: bool

Default: false

kafka.tls.enabled

Whether custom TLS settings are enabled.

Type: bool

Default: false

kafka.tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or root_cas_file for file-based certificate loading.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

kafka.tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or root_cas for inline certificate data.

Type: string

Default: ""

# Examples:
root_cas_file: ./root_cas.pem

kafka.tls.skip_cert_verify

Whether to skip server-side certificate verification. Set to true only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using root_cas or root_cas_file to specify trusted certificates instead of disabling verification entirely.

Type: bool

Default: false

kafka.topic_lag_refresh_period

The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as 100ms, 1s, or 5s.

Type: string

Default: 5s

kafka.topics[]

A list of topics to consume from (required). You can list multiple comma-separated topics in a single element.

If you specify a consumer_group, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed.

Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, foo:0 consumes the partition 0 of the topic foo. This syntax also supports ranges. For example, foo:0-10 consumes all partitions from 0 through to 10 inclusively.

Finally, add another colon after the partition to set an explicit offset to consume from. For example, foo:0:10 consumes the partition 0 of the topic foo starting from the offset 10. If the offset is not present (or remains unspecified) then the field start_offset determines which offset to start from.

Type: array

# Examples:
topics:
  - foo
  - bar

# ---

topics:
  - things.*

# ---

topics:
  - "foo,bar"

# ---

topics:
  - "foo:0"
  - "bar:1"
  - "bar:3"

# ---

topics:
  - "foo:0,bar:1,bar:3"

# ---

topics:
  - "foo:0-5"

kafka.transaction_isolation_level

The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees.

Type: string

Default: read_uncommitted

Option Summary

read_committed

If set, only committed transactional records are processed.

read_uncommitted

If set, then uncommitted records are processed.

node_address

The TCP listening address of the Ockam node.

Type: string

Default: 127.0.0.1:6262

relay

Make the Ockam node accessible through a relay with the supplied name (optional).

For example, setting this value to orders_consumer would require you to set the route_to_consumer on any producer to /project/default/service/forward_to_orders_consumer/secure/api.

Type: string

route_to_kafka_outlet

The route to reach the Kafka Portal Outlet of your Ockam portal. For example, /project/default.

Type: string

Default: self