ockam_kafka

Beta

Type:

Available in: Self-Managed

Uses Ockam to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the ockam_kafka output or by creating a Kafka Portal Inlet using Ockam Command.

Ockam Secure Channels guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages.

You can use Ockam to encrypt whole messages, or specific fields in a message.

Introduced in version 4.33.0.

  • Common

  • Advanced

# Common configuration fields, showing default values
input:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      topics: [] # No default (required)
      regexp_topics: false
      consumer_group: "" # No default (optional)
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
# All configuration fields, showing default values
input:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      topics: [] # No default (required)
      regexp_topics: false
      rack_id: ""
      start_from_oldest: true
      fetch_max_bytes: 50MiB
      fetch_max_wait: 5s
      fetch_min_bytes: 1B
      fetch_max_partition_bytes: 1MiB
      consumer_group: "" # No default (optional)
      checkpoint_limit: 1024
      commit_period: 5s
      multi_header: false
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      disable_content_encryption: false
      enrollment_ticket: "" # No default (optional)
      identity_name: "" # No default (optional)
      allow: self
      route_to_kafka_outlet: self
      allow_producer: self
      relay: "" # No default (optional)
      node_address: 127.0.0.1:6262
      encrypted_fields: []

Fields

kafka.seed_brokers

A list of broker addresses to connect to (optional). List items that contain commas are expanded into multiple addresses.

Type: array

# Examples

seed_brokers:
  - localhost:9092

seed_brokers:
  - foo:9092
  - bar:9092

seed_brokers:
  - foo:9092,bar:9092

kafka.tls

Override system defaults with custom TLS settings.

Type: object

kafka.tls.enabled

Whether custom TLS settings are enabled.

Type: bool

Default: false

kafka.tls.skip_cert_verify

Whether to skip server-side certificate verification.

Type: bool

Default: false

kafka.tls.enable_renegotiation

Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation.

Type: bool

Default: false

Requires version 3.45.0 or newer

kafka.tls.root_cas

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples

root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

kafka.tls.root_cas_file

Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate.

Type: string

Default: ""

# Examples

root_cas_file: ./root_cas.pem

kafka.tls.client_certs

A list of client certificates to use. For each certificate, specify either the fields cert and key or cert_file and key_file.

Type: array

Default: []

# Examples

client_certs:
  - cert: foo
    key: bar

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key

kafka.tls.client_certs[].cert

A plain text certificate to use.

Type: string

Default: ""

kafka.tls.client_certs[].key

The plain text certificate key to use.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

kafka.tls.client_certs[].cert_file

The path of a certificate to use.

Type: string

Default: ""

kafka.tls.client_certs[].key_file

The path of a certificate key to use.

Type: string

Default: ""

kafka.tls.client_certs[].password

The plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete pbeWithMD5AndDES-CBC algorithm is not supported for the PKCS#8 format.

The pbeWithMD5AndDES-CBC algorithm does not authenticate ciphertext, and is vulnerable to padding oracle attacks that may allow an attacker to recover the plain text password.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

# Examples

password: foo

password: ${KEY_PASSWORD}

kafka.topics

A list of topics to consume from (required). You can list multiple comma-separated topics in a single element.

If you specify a consumer_group, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed.

Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, foo:0 consumes the partition 0 of the topic foo. This syntax also supports ranges. For example, foo:0-10 consumes all partitions from 0 through to 10 inclusively.

Finally, add another colon after the partition to set an explicit offset to consume from. For example, foo:0:10 consumes the partition 0 of the topic foo starting from the offset 10. If the offset is not present (or remains unspecified) then the field start_from_oldest determines which offset to start from.

Type: array

# Examples

topics:
  - foo
  - bar

topics:
  - things.*

topics:
  - foo,bar

topics:
  - foo:0
  - bar:1
  - bar:3

topics:
  - foo:0,bar:1,bar:3

topics:
  - foo:0-5

kafka.regexp_topics

Whether listed topics are interpreted as regular expression patterns for matching multiple topics. Set this value to false when explicit partitions are specified for topics.

Type: bool

Default: false

kafka.rack_id

A rack identifier for this client.

Type: string

Default: ""

kafka.start_from_oldest

Whether to consume from the oldest available offset. Otherwise, messages are consumed from the latest offset. This setting is applied when creating a new consumer group or the saved offset no longer exists.

Type: bool

Default: true

fetch_max_bytes

The maximum size of a message batch (in bytes) that a broker tries to send during a client fetch. If individual records exceed the fetch_max_bytes value, brokers will still send them.

Type: string

Default: 50MiB

fetch_max_wait

The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (fetch_min_bytes).

Type: string

Default: 5s

fetch_min_bytes

The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting fetch.min.bytes.

Type: string

Default: 1B

fetch_max_partition_bytes

The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting fetch.max.partition.bytes.

If a single batch is larger than the fetch_max_partition_bytes value, the batch is still sent so that the client can make progress.

Type: string

Default: 1MiB

kafka.consumer_group

Assign a consumer group for the processing of messages (optional). When this value is set:

  • Partitions of specified topics are automatically distributed across consumers sharing a consumer group.

  • Partition offsets are automatically committed and resumed under this name.

Consumer groups are not supported when explicit partitions to consume from are specified in the topics field.

Type: string

kafka.checkpoint_limit

The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied.

When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the checkpoint_limit value.

Type: int

Default: 1024

kafka.commit_period

The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown.

Type: string

Default: 5s

kafka.multi_header

Decode headers into lists to allow the handling of multiple values with the same key.

Type: bool

Default: false

kafka.batching

Configure a batching policy for individual topic partitions. This allows the input to batch messages together before flushing them for processing. Batching may improve performance and is useful for windowed processing as it preserves the ordering of topic partitions.

Type: object

# Examples

batching:
  byte_size: 5000
  count: 0
  period: 1s

batching:
  count: 10
  period: 1s

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

kafka.batching.count

The number of messages after which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

kafka.batching.byte_size

The number of bytes at which the batch is flushed. Set to 0 to disable size-based batching.

Type: int

Default: 0

kafka.batching.period

The period of time after which an incomplete batch is flushed regardless of its size.

Type: string

Default: ""

# Examples

period: 1s

period: 1m

period: 500ms

kafka.batching.check

A Bloblang query that returns a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples

check: this.type == "end_of_transaction"

kafka.batching.processors

For aggregating and archiving message batches, you can add a list of processors to apply to a batch as it is flushed (optional). All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches.

Type: array

# Examples

processors:
  - archive:
      format: concatenate

processors:
  - archive:
      format: lines

processors:
  - archive:
      format: json_array

disable_content_encryption

Disables Kafka message encryption.

If this value is set to true:

  • Only message payloads remain unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled.

  • All other ockam_kafka inlets and outlets in a topic must also have their settings set to true.

Type: bool

Default: false

enrollment_ticket

The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional).

You can generate a new ticket using the ockam project ticket command.

Type: string

identity_name

The name of the Ockam identity to use. If this value is not provided, the default Ockam identity is automatically generated and used (optional).

Type: string

allow

Use in conjunction with the route_to_kafka_outlet field to specify an access control policy for the Kafka Portal Outlet.

For example, setting this value to kafka_us_east forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute kafka_us_east=true.

Type: string

Default: self

route_to_kafka_outlet

The route to reach the Kafka Portal Outlet of your Ockam portal. For example, /project/default.

Type: string

Default: self

allow_producer

Specify an access control policy for producers.

For example, setting this value to orders_producer forces the producer to present an Ockam credential, which confirms that the producer has the attribute orders_producer=true.

Type: string

Default: self

relay

Make the Ockam node accessible through a relay with the supplied name (optional).

For example, setting this value to orders_consumer would require you to set the route_to_consumer on any producer to /project/default/service/forward_to_orders_consumer/secure/api.

Type: string

node_address

The TCP listening address of the Ockam node.

Type: string

Default: 127.0.0.1:6262

encrypted_fields

The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted.

Type: string

Default: []