Docs Connect Components Inputs ockam_kafka ockam_kafka Beta Type: InputOutput Available in: Self-Managed Uses Ockam to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the ockam_kafka output or by creating a Kafka Portal Inlet using Ockam Command. Ockam Secure Channels guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages. You can use Ockam to encrypt whole messages, or specific fields in a message. Introduced in version 4.33.0. Common Advanced # Common configuration fields, showing default values input: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) topics: [] # No default (required) regexp_topics: false consumer_group: "" # No default (optional) auto_replay_nacks: true disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_producer: self relay: "" # No default (optional) node_address: 127.0.0.1:6262 # All configuration fields, showing default values input: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) topics: [] # No default (required) regexp_topics: false consumer_group: "" # No default (optional) client_id: benthos rack_id: "" checkpoint_limit: 1024 auto_replay_nacks: true commit_period: 5s start_from_oldest: true metadata_max_age: 5m tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_producer: self relay: "" # No default (optional) node_address: 127.0.0.1:6262 Fields kafka.seed_brokers A list of broker addresses to connect to (optional). List items that contain commas are expanded into multiple addresses. Type: array # Examples seed_brokers: - localhost:9092 seed_brokers: - foo:9092 - bar:9092 seed_brokers: - foo:9092,bar:9092 kafka.topics A list of topics to consume from (required). You can list multiple comma-separated topics in a single element. If you specify a consumer_group, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, foo:0 consumes the partition 0 of the topic foo. This syntax also supports ranges. For example, foo:0-10 consumes all partitions from 0 through to 10 inclusively. Finally, add another colon after the partition to set an explicit offset to consume from. For example, foo:0:10 consumes the partition 0 of the topic foo starting from the offset 10. If the offset is not present (or remains unspecified) then the field start_from_oldest determines which offset to start from. Type: array # Examples topics: - foo - bar topics: - things.* topics: - foo,bar topics: - foo:0 - bar:1 - bar:3 topics: - foo:0,bar:1,bar:3 topics: - foo:0-5 kafka.regexp_topics Whether listed topics are interpreted as regular expression patterns for matching multiple topics. Set this value to false when explicit partitions are specified for topics. Type: bool Default: false kafka.consumer_group Assign a consumer group for the processing of messages (optional). When this value is set: Partitions of specified topics are automatically distributed across consumers sharing a consumer group. Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when explicit partitions to consume from are specified in the topics field. Type: string kafka.client_id An identifier for the client connection. Type: string Default: benthos kafka.rack_id A rack identifier for this client. Type: string Default: "" kafka.checkpoint_limit The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the checkpoint_limit value. Type: int Default: 1024 kafka.auto_replay_nacks Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of message rejections is persistent, replaying them may cause back pressure. Set this value to false to delete rejected messages. Disabling automatic replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is immediately discarded upon consumption and mutation. Type: bool Default: true kafka.commit_period The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. Type: string Default: 5s kafka.start_from_oldest Whether to consume from the oldest available offset. Otherwise, messages are consumed from the latest offset. This setting is applied when creating a new consumer group or the saved offset no longer exists. Type: bool Default: true kafka.metadata_max_age The maximum period of time after which metadata is refreshed. Type: string Default: 5m kafka.tls Override system defaults with custom TLS settings. Type: object kafka.tls.enabled Whether custom TLS settings are enabled. Type: bool Default: false kafka.tls.skip_cert_verify Whether to skip server-side certificate verification. Type: bool Default: false kafka.tls.enable_renegotiation Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation. Type: bool Default: false Requires version 3.45.0 or newer kafka.tls.root_cas Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets. Type: string Default: "" # Examples root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- kafka.tls.root_cas_file Specify the path to a root certificate authority file (optional). This is a file, often with a .pem extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Type: string Default: "" # Examples root_cas_file: ./root_cas.pem kafka.tls.client_certs A list of client certificates to use. For each certificate, specify either the fields cert and key or cert_file and key_file. Type: array Default: [] # Examples client_certs: - cert: foo key: bar client_certs: - cert_file: ./example.pem key_file: ./example.key kafka.tls.client_certs[].cert A plain text certificate to use. Type: string Default: "" kafka.tls.client_certs[].key The plain text certificate key to use. This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets. Type: string Default: "" kafka.tls.client_certs[].cert_file The path of a certificate to use. Type: string Default: "" kafka.tls.client_certs[].key_file The path of a certificate key to use. Type: string Default: "" kafka.tls.client_certs[].password The plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete pbeWithMD5AndDES-CBC algorithm is not supported for the PKCS#8 format. The pbeWithMD5AndDES-CBC algorithm does not authenticate ciphertext, and is vulnerable to padding oracle attacks that may allow an attacker recover the plain text password. This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets. Type: string Default: "" # Examples password: foo password: ${KEY_PASSWORD} kafka.sasl Specify one or more methods or mechanisms of SASL authentication, which are attempted in order (optional). If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. Type: array # Examples sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo kafka.sasl[].mechanism The SASL mechanism to use. Type: string Option Summary AWS_MSK_IAM AWS IAM-based authentication as specified by the aws-msk-iam-auth Java library. OAUTHBEARER OAuth bearer-based authentication. PLAIN Plain text authentication. SCRAM-SHA-256 SCRAM-based authentication as specified in RFC5802. SCRAM-SHA-512 SCRAM-based authentication as specified in RFC5802. none Disable SASL authentication kafka.sasl[].username A username for PLAIN or SCRAM-* authentication. Type: string Default: "" kafka.sasl[].password A password for PLAIN or SCRAM-* authentication. This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets. Type: string Default: "" kafka.sasl[].token The token to use for a single session’s OAUTHBEARER authentication. Type: string Default: "" kafka.sasl[].extensions Key/value pairs to add to OAUTHBEARER authentication requests. Type: object kafka.sasl[].aws AWS specific fields for when the mechanism is set to AWS_MSK_IAM. Type: object kafka.sasl[].aws.region The AWS region to target. Type: string Default: "" kafka.sasl[].aws.endpoint Specify a custom endpoint for the AWS API. Type: string Default: "" kafka.sasl[].aws.credentials Manually configure the AWS credentials to use (optional). For more information, see the Amazon Web Services guide. Type: object kafka.sasl[].aws.credentials.profile The profile from ~/.aws/credentials to use. Type: string Default: "" kafka.sasl[].aws.credentials.id The ID of credentials to use. Type: string Default: "" kafka.sasl[].aws.credentials.secret The secret for the AWS credentials in use. This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets. Type: string Default: "" kafka.sasl[].aws.credentials.token The token for the AWS credentials in use. This is a required value for short-term credentials. Type: string Default: "" kafka.sasl[].aws.credentials.from_ec2_role Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance. Type: bool Default: false Requires version 4.2.0 or newer kafka.sasl[].aws.credentials.role The role ARN to assume. Type: string Default: "" kafka.sasl[].aws.credentials.role_external_id An external ID to use when assuming a role. Type: string Default: "" kafka.multi_header Decode headers into lists to allow the handling of multiple values with the same key. Type: bool Default: false kafka.batching Configure a batching policy for individual topic partitions. This allows the input to batch messages together before flushing them for processing. Batching may improve performance and is useful for windowed processing as it preserves the ordering of topic partitions. Type: object # Examples batching: byte_size: 5000 count: 0 period: 1s batching: count: 10 period: 1s batching: check: this.contains("END BATCH") count: 0 period: 1m kafka.batching.count The number of messages after which the batch is flushed. Set to 0 to disable count-based batching. Type: int Default: 0 kafka.batching.byte_size The number of bytes at which the batch is flushed. Set to 0 to disable size-based batching. Type: int Default: 0 kafka.batching.period The period of time after which an incomplete batch is flushed regardless of its size. Type: string Default: "" # Examples period: 1s period: 1m period: 500ms kafka.batching.check A Bloblang query that returns a boolean value indicating whether a message should end a batch. Type: string Default: "" # Examples check: this.type == "end_of_transaction" kafka.batching.processors For aggregating and archiving message batches, you can add a list of processors to apply to a batch as it is flushed (optional). All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches. Type: array # Examples processors: - archive: format: concatenate processors: - archive: format: lines processors: - archive: format: json_array disable_content_encryption Disables Kafka message encryption. If this value is set to true: Only message payloads remain unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled. All other ockam_kafka inlets and outlets in a topic must also have their settings set to true. Type: bool Default: false enrollment_ticket The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional). You can generate a new ticket using the ockam project ticket command. Type: string identity_name The name of the Ockam identity to use. If this value is not provided, the default Ockam identity is automatically generated and used (optional). Type: string allow Use in conjunction with the route_to_kafka_outlet field to specify an access control policy for the Kafka Portal Outlet. For example, setting this value to kafka_us_east forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute kafka_us_east=true. Type: string Default: self route_to_kafka_outlet The route to reach the Kafka Portal Outlet of your Ockam portal. For example, /project/default. Type: string Default: self allow_producer Specify an access control policy for producers. For example, setting this value to orders_producer forces the producer to present an Ockam credential, which confirms that the producer has the attribute orders_producer=true. Type: string Default: self relay Make the Ockam node accessible through a relay with the supplied name (optional). For example, setting this value to orders_consumer would require you to set the route_to_consumer on any producer to /project/default/service/forward_to_orders_consumer/secure/api. Type: string node_address The TCP listening address of the Ockam node. Type: string Default: 127.0.0.1:6262 Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution nsq parquet