# ockam_kafka

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [connect-full.txt](https://docs.redpanda.com/connect-full.txt)

---
title: ockam_kafka
latest-connect-version: 4.93.0
latest-operator-version: v26.1.4
latest-console-tag: v3.7.3
latest-redpanda-tag: v26.1.9
docname: inputs/ockam_kafka
page-component-name: connect
page-version: master
page-component-version: master
page-component-title: Connect
page-relative-src-path: inputs/ockam_kafka.adoc
page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/ockam_kafka.adoc
page-git-created-date: "2024-11-25"
page-git-modified-date: "2026-05-26"
---

<!-- Source: https://docs.redpanda.com/connect/components/inputs/ockam_kafka.md -->

**Type:** Input ▼

[Input](https://docs.redpanda.com/connect/components/inputs/ockam_kafka/)[Output](https://docs.redpanda.com/connect/components/outputs/ockam_kafka/)

**Available in:** Self-Managed

Uses [Ockam](https://docs.ockam.io/) to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the `ockam_kafka` output or by [creating a Kafka Portal Inlet](https://command.ockam.io/manual/ockam-kafka-inlet-create.html) using Ockam Command.

[Ockam Secure Channels](https://docs.ockam.io/reference/command/secure-channels) guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages.

> 📝 **NOTE**
>
> You can use Ockam to encrypt whole messages, or specific fields in a message.

Introduced in version 4.33.0.

#### Common

```yml
inputs:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      topics: [] # No default (optional)
      regexp_topics_include: [] # No default (optional)
      regexp_topics_exclude: [] # No default (optional)
      rack_id: ""
      instance_id: ""
      rebalance_timeout: 45s
      session_timeout: 1m
      heartbeat_interval: 3s
      start_offset: earliest
      fetch_max_bytes: 50MiB
      fetch_max_wait: 5s
      fetch_min_bytes: 1B
      fetch_max_partition_bytes: 1MiB
      transaction_isolation_level: read_uncommitted
      consumer_group: "" # No default (optional)
      checkpoint_limit: 1024
      commit_period: 5s
      multi_header: false
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      topic_lag_refresh_period: 5s
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
    encrypted_fields: []
```

#### Advanced

```yml
inputs:
  label: ""
  ockam_kafka:
    kafka:
      seed_brokers: [] # No default (optional)
      tls:
        enabled: false
        skip_cert_verify: false
        enable_renegotiation: false
        root_cas: ""
        root_cas_file: ""
        client_certs: []
      topics: [] # No default (optional)
      regexp_topics_include: [] # No default (optional)
      regexp_topics_exclude: [] # No default (optional)
      rack_id: ""
      instance_id: ""
      rebalance_timeout: 45s
      session_timeout: 1m
      heartbeat_interval: 3s
      start_offset: earliest
      fetch_max_bytes: 50MiB
      fetch_max_wait: 5s
      fetch_min_bytes: 1B
      fetch_max_partition_bytes: 1MiB
      transaction_isolation_level: read_uncommitted
      consumer_group: "" # No default (optional)
      checkpoint_limit: 1024
      commit_period: 5s
      multi_header: false
      batching:
        count: 0
        byte_size: 0
        period: ""
        check: ""
        processors: [] # No default (optional)
      topic_lag_refresh_period: 5s
    disable_content_encryption: false
    enrollment_ticket: "" # No default (optional)
    identity_name: "" # No default (optional)
    allow: self
    route_to_kafka_outlet: self
    allow_producer: self
    relay: "" # No default (optional)
    node_address: 127.0.0.1:6262
    encrypted_fields: []
```

## [](#fields)Fields

### [](#allow)`allow`

Use in conjunction with the `route_to_kafka_outlet` field to specify an access control policy for the Kafka Portal Outlet.

For example, setting this value to `kafka_us_east` forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute `kafka_us_east=true`.

**Type**: `string`

**Default**: `self`

### [](#allow_producer)`allow_producer`

Specify an access control policy for producers.

For example, setting this value to `orders_producer` forces the producer to present an Ockam credential, which confirms that the producer has the attribute `orders_producer=true`.

**Type**: `string`

**Default**: `self`

### [](#disable_content_encryption)`disable_content_encryption`

Disables Kafka message encryption.

If this value is set to `true`:

-   Only message payloads remain unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled.

-   All other `ockam_kafka` inlets and outlets in a topic must also have their settings set to `true`.


**Type**: `bool`

**Default**: `false`

### [](#encrypted_fields)`encrypted_fields[]`

The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted.

**Type**: `array`

**Default**: `[]`

### [](#enrollment_ticket)`enrollment_ticket`

The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional).

You can generate a new ticket using the [`ockam project ticket` command](https://command.ockam.io/manual/ockam-project-ticket.html).

**Type**: `string`

### [](#identity_name)`identity_name`

The name of the [Ockam identity](https://command.ockam.io/manual/ockam-identity.html) to use. If this value is not provided, the default Ockam identity is automatically generated and used (optional).

**Type**: `string`

### [](#kafka)`kafka`

**Type**: `object`

### [](#kafka-batching)`kafka.batching`

Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/) for individual topic partitions. This allows the input to batch messages together before flushing them for processing. Batching may improve performance and is useful for windowed processing as it preserves the ordering of topic partitions.

**Type**: `object`

```yaml
# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m
```

### [](#kafka-batching-byte_size)`kafka.batching.byte_size`

The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching.

**Type**: `int`

**Default**: `0`

### [](#kafka-batching-check)`kafka.batching.check`

A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch.

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
check: this.type == "end_of_transaction"
```

### [](#kafka-batching-count)`kafka.batching.count`

The number of messages after which the batch is flushed. Set to `0` to disable count-based batching.

**Type**: `int`

**Default**: `0`

### [](#kafka-batching-period)`kafka.batching.period`

The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`.

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms
```

### [](#kafka-batching-processors)`kafka.batching.processors[]`

For aggregating and archiving message batches, you can add a list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed (optional). All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches.

**Type**: `processor`

```yaml
# Examples:
processors:
  - archive:
      format: concatenate

# ---

processors:
  - archive:
      format: lines

# ---

processors:
  - archive:
      format: json_array
```

### [](#kafka-checkpoint_limit)`kafka.checkpoint_limit`

The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied.

When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value.

**Type**: `int`

**Default**: `1024`

### [](#kafka-commit_period)`kafka.commit_period`

The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown.

**Type**: `string`

**Default**: `5s`

### [](#kafka-consumer_group)`kafka.consumer_group`

Assign a consumer group for the processing of messages (optional). When this value is set:

-   Partitions of specified topics are automatically distributed across consumers sharing a consumer group.

-   Partition offsets are automatically committed and resumed under this name.


Consumer groups are not supported when explicit partitions to consume from are specified in the `topics` field.

**Type**: `string`

### [](#kafka-fetch_max_bytes)`kafka.fetch_max_bytes`

The maximum size of a message batch (in bytes) that a broker tries to send during a client fetch. If individual records exceed the `fetch_max_bytes` value, brokers will still send them.

**Type**: `string`

**Default**: `50MiB`

### [](#kafka-fetch_max_partition_bytes)`kafka.fetch_max_partition_bytes`

The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`.

If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress.

**Type**: `string`

**Default**: `1MiB`

### [](#kafka-fetch_max_wait)`kafka.fetch_max_wait`

The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`).

**Type**: `string`

**Default**: `5s`

### [](#kafka-fetch_min_bytes)`kafka.fetch_min_bytes`

The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`.

**Type**: `string`

**Default**: `1B`

### [](#kafka-heartbeat_interval)`kafka.heartbeat_interval`

When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active.

You must set `heartbeat_interval` to less than one-third of `session_timeout`.

This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`.

**Type**: `string`

**Default**: `3s`

### [](#kafka-instance_id)`kafka.instance_id`

When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections.

When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID.

**Type**: `string`

**Default**: `""`

### [](#kafka-multi_header)`kafka.multi_header`

Decode headers into lists to allow the handling of multiple values with the same key.

**Type**: `bool`

**Default**: `false`

### [](#kafka-rack_id)`kafka.rack_id`

A rack identifier for this client.

**Type**: `string`

**Default**: `""`

### [](#kafka-rebalance_timeout)`kafka.rebalance_timeout`

When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`.

**Type**: `string`

**Default**: `45s`

### [](#kafka-regexp_topics_exclude)`kafka.regexp_topics_exclude[]`

A list of regular expression patterns for excluding topics when regex mode is enabled (via `regexp_topics` or `regexp_topics_include`). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns.

**Type**: `array`

### [](#kafka-regexp_topics_include)`kafka.regexp_topics_include[]`

A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. This enables regex mode and cannot be used together with the `topics` field. Use `regexp_topics_exclude` to exclude specific patterns.

**Type**: `array`

```yaml
# Examples:
regexp_topics_include:
  - logs_.*
  - metrics_.*

# ---

regexp_topics_include:
  - "events_[0-9]+"
```

### [](#kafka-seed_brokers)`kafka.seed_brokers[]`

A list of broker addresses to connect to (optional). List items that contain commas are expanded into multiple addresses.

**Type**: `array`

```yaml
# Examples:
seed_brokers:
  - "localhost:9092"

# ---

seed_brokers:
  - "foo:9092"
  - "bar:9092"

# ---

seed_brokers:
  - "foo:9092,bar:9092"
```

### [](#kafka-session_timeout)`kafka.session_timeout`

When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`.

**Type**: `string`

**Default**: `1m`

### [](#kafka-start_offset)`kafka.start_offset`

Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch.

**Type**: `string`

**Default**: `earliest`

| Option | Summary |
| --- | --- |
| committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option |
| earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. |
| latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. |

### [](#kafka-tls)`kafka.tls`

Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments.

**Type**: `object`

### [](#kafka-tls-client_certs)`kafka.tls.client_certs[]`

A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates.

You must set `tls.enabled: true` for the client certificates to take effect.

**Certificate pairing rules**: For each certificate item, provide either:

-   Inline PEM data using both `cert` **and** `key` or

-   File paths using both `cert_file` **and** `key_file`.


Mixing inline and file-based values within the same item is not supported.

**Type**: `object`

**Default**: `[]`

```yaml
# Examples:
client_certs:
  - cert: foo
    key: bar

# ---

client_certs:
  - cert_file: ./example.pem
    key_file: ./example.key
```

### [](#kafka-tls-client_certs-cert)`kafka.tls.client_certs[].cert`

A plain text certificate to use.

**Type**: `string`

**Default**: `""`

### [](#kafka-tls-client_certs-cert_file)`kafka.tls.client_certs[].cert_file`

The path of a certificate to use.

**Type**: `string`

**Default**: `""`

### [](#kafka-tls-client_certs-key)`kafka.tls.client_certs[].key`

A plain text certificate key to use.

> ⚠️ **CAUTION**
>
> This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/).

**Type**: `string`

**Default**: `""`

### [](#kafka-tls-client_certs-key_file)`kafka.tls.client_certs[].key_file`

The path of a certificate key to use.

**Type**: `string`

**Default**: `""`

### [](#kafka-tls-client_certs-password)`kafka.tls.client_certs[].password`

A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format.

Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext.

> ⚠️ **CAUTION**
>
> This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/).

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
password: foo

# ---

password: ${KEY_PASSWORD}
```

### [](#kafka-tls-enable_renegotiation)`kafka.tls.enable_renegotiation`

Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`.

Requires version 3.45.0 or later.

**Type**: `bool`

**Default**: `false`

### [](#kafka-tls-enabled)`kafka.tls.enabled`

Whether custom TLS settings are enabled.

**Type**: `bool`

**Default**: `false`

### [](#kafka-tls-root_cas)`kafka.tls.root_cas`

Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading.

> ⚠️ **CAUTION**
>
> This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/).

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
root_cas: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
```

### [](#kafka-tls-root_cas_file)`kafka.tls.root_cas_file`

Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data.

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
root_cas_file: ./root_cas.pem
```

### [](#kafka-tls-skip_cert_verify)`kafka.tls.skip_cert_verify`

Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely.

**Type**: `bool`

**Default**: `false`

### [](#kafka-topic_lag_refresh_period)`kafka.topic_lag_refresh_period`

The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`.

**Type**: `string`

**Default**: `5s`

### [](#kafka-topics)`kafka.topics[]`

A list of topics to consume from (required). You can list multiple comma-separated topics in a single element.

If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed.

Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusively.

Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from.

**Type**: `array`

```yaml
# Examples:
topics:
  - foo
  - bar

# ---

topics:
  - things.*

# ---

topics:
  - "foo,bar"

# ---

topics:
  - "foo:0"
  - "bar:1"
  - "bar:3"

# ---

topics:
  - "foo:0,bar:1,bar:3"

# ---

topics:
  - "foo:0-5"
```

### [](#kafka-transaction_isolation_level)`kafka.transaction_isolation_level`

The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees.

**Type**: `string`

**Default**: `read_uncommitted`

| Option | Summary |
| --- | --- |
| read_committed | If set, only committed transactional records are processed. |
| read_uncommitted | If set, then uncommitted records are processed. |

### [](#node_address)`node_address`

The TCP listening address of the Ockam node.

**Type**: `string`

**Default**: `127.0.0.1:6262`

### [](#relay)`relay`

Make the Ockam node accessible through a relay with the supplied name (optional).

For example, setting this value to `orders_consumer` would require you to set the `route_to_consumer` on any producer to `/project/default/service/forward_to_orders_consumer/secure/api`.

**Type**: `string`

### [](#route_to_kafka_outlet)`route_to_kafka_outlet`

The route to reach the Kafka Portal Outlet of your Ockam portal. For example, `/project/default`.

**Type**: `string`

**Default**: `self`