redpanda_migrator_offsets
beta
Reads consumer group offsets for a specified set of topics using the Franz Kafka client library.
-
Common
-
Advanced
# Common configuration fields, showing default values
input:
label: ""
redpanda_migrator_offsets:
seed_brokers: [] # No default (required)
topics: [] # No default (required)
regexp_topics: false
consumer_group: "" # No default (optional)
auto_replay_nacks: true
# All configuration fields, showing default values
input:
label: ""
redpanda_migrator_offsets:
seed_brokers: [] # No default (required)
client_id: benthos
tls:
enabled: false
skip_cert_verify: false
enable_renegotiation: false
root_cas: ""
root_cas_file: ""
client_certs: []
sasl: [] # No default (optional)
metadata_max_age: 5m
topics: [] # No default (required)
regexp_topics: false
rack_id: ""
consumer_group: "" # No default (optional)
commit_period: 5s
partition_buffer_bytes: 1MB
topic_lag_refresh_period: 5s
auto_replay_nacks: true
Metadata
This input adds the following metadata fields to each message:
-
kafka_key
-
kafka_topic
-
kafka_partition
-
kafka_offset
-
kafka_timestamp_unix
-
kafka_timestamp_ms
-
kafka_tombstone_message
-
kafka_offset_topic
-
kafka_offset_group
-
kafka_offset_partition
-
kafka_offset_commit_timestamp
-
kafka_offset_metadata
-
kafka_is_high_watermark
Fields
seed_brokers
A list of broker addresses to connect to. Use commas to separate multiple addresses in a single list item.
Type: array
# Examples
seed_brokers:
- localhost:9092
seed_brokers:
- foo:9092
- bar:9092
seed_brokers:
- foo:9092,bar:9092
tls.skip_cert_verify
Whether to skip server-side certificate verification.
Type: bool
Default: false
tls.enable_renegotiation
Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message local error: tls: no renegotiation
.
Type: bool
Default: false
tls.root_cas
Specify a certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
# Examples
root_cas: |-
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.root_cas_file
Specify the path to a root certificate authority file (optional). This is a file, often with a .pem
extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate.
Type: string
Default: ""
# Examples
root_cas_file: ./root_cas.pem
tls.client_certs
A list of client certificates to use. For each certificate, specify values for either the cert
and key
, or cert_file
and key_file
fields.
Type: array
Default: []
# Examples
client_certs:
- cert: foo
key: bar
client_certs:
- cert_file: ./example.pem
key_file: ./example.key
tls.client_certs[].key
A plain text certificate key to use.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
tls.client_certs[].password
A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete pbeWithMD5AndDES-CBC
algorithm is not supported for the PKCS#8 format.
The pbeWithMD5AndDES-CBC algorithm does not authenticate ciphertext, and is vulnerable to padding Oracle attacks which may allow an attacker recover the plain text password.
|
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
# Examples
password: foo
password: ${KEY_PASSWORD}
sasl
Specify one or more methods of SASL authentication, which are attempted in order. If the broker supports the first mechanism, all connections use that mechanism. If the first mechanism fails, the client picks the first supported mechanism. Connections fail if the broker does not support any client mechanisms.
Type: array
# Examples
sasl:
- mechanism: SCRAM-SHA-512
password: bar
username: foo
sasl[].mechanism
The SASL mechanism to use.
Type: string
Option | Summary |
---|---|
|
AWS IAM-based authentication as specified by the |
|
OAuth bearer-based authentication. |
|
Plain text authentication. |
|
SCRAM-based authentication as specified in RFC 5802. |
|
SCRAM-based authentication as specified in RFC 5802. |
|
Disable SASL authentication. |
sasl[].password
A password to provide for PLAIN
or SCRAM-*
authentication.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
sasl[].token
The token to use for a single session’s OAUTHBEARER
authentication.
Type: string
Default: ""
sasl[].aws.credentials
Manually configure the AWS credentials to use (optional). For more information, see the Amazon Web Services guide.
Type: object
sasl[].aws.credentials.secret
The secret for the credentials being used.
This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration. |
Type: string
Default: ""
sasl[].aws.credentials.token
The token for the credentials being used, required when using short-term credentials.
Type: string
Default: ""
sasl[].aws.credentials.from_ec2_role
Use the credentials of a host EC2 machine configured to assume an IAM role associated with the instance.
Type: bool
Default: false
sasl[].aws.credentials.role_external_id
An external ID to provide when assuming a role.
Type: string
Default: ""
metadata_max_age
The maximum period of time (in minutes) after which metadata is refreshed.
Type: string
Default: 5m
topics
A list of topics to consume from. Use commas to separate multiple topics in a single element.
When a consumer_group
is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed.
Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, foo:0
would consume the partition 0
of the topic foo. This syntax supports ranges. For example, foo:0-10
would consume partitions 0
through to 10
inclusive.
It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, foo:0:10
would consume the partition 0
of the topic foo
starting from the offset 10
. If the offset is not present (or remains unspecified) then the field start_from_oldest
determines which offset to start from.
Type: array
# Examples
topics:
- foo
- bar
topics:
- things.*
topics:
- foo,bar
regexp_topics
Whether listed topics are interpreted as regular expression patterns for matching multiple topics. When topics are specified with explicit partitions, this field must remain set to false
.
Type: bool
Default: false
rack_id
A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica.
Type: string
Default: ""
consumer_group
An optional consumer group. When this value is specified:
-
The partitions of any topics, specified in the
topics
field, are automatically distributed across consumers sharing a consumer group -
Partition offsets are automatically committed and resumed under this name
Consumer groups are not supported when you specify explicit partitions to consume from in the topics
field.
Type: string
commit_period
The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown.
Type: string
Default: 5s
partition_buffer_bytes
A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization.
Each buffer can grow slightly beyond this value.
Type: string
Default: 1MB
topic_lag_refresh_period
The interval between refresh cycles. During each cycle, this input queries the Repanda Connect server to calculate the topic lag - the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group.
Type: string
Default: 5s
auto_replay_nacks
Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure.
Set auto_replay_nacks
to false
to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation.
Type: bool
Default: true