Create a GCS Sink Connector

File name template

file.name.template

File name prefix

file.name.prefix

The prefix to be added to the name of each file put in GCS.

Output fields

format.output.fields

Fields to place into output files. Supported values are: 'key', 'value', 'offset', 'timestamp', and 'headers'.

Value field encoding

format.output.fields.value.encoding

The type of encoding to be used for the value field. Supported values are: 'none' and 'base64'.

Envelope for primitives

format.output.envelope

Specifies whether or not to enable additional JSON object wrapping of the actual value.

Output file compression

file.compression.type

The compression type to be used for files put into GCS. Supported values are: 'none', 'gzip', 'snappy', and 'zstd'.

Max records per file

file.max.records

The maximum number of records to put in a single file. Must be a non-negative number. 0 is interpreted as "unlimited", which is the default. In this case files are only flushed after file.flush.interval.ms.

File flush interval milliseconds

file.flush.interval.ms

The time interval to periodically flush files and commit offsets. Value specified must be a non-negative number. Default is 60 seconds. 0 indicates that it is disabled. In this case, files are only flushed after reaching file.max.records record size.

GCS bucket check

gcs.bucket.check

If set to true, the connector will attempt to put a test file to the GCS bucket to validate access. Default is true.

GCS retry backoff initial delay milliseconds

gcs.retry.backoff.initial.delay.ms

Initial retry delay in milliseconds. The default value is 1000.

GCS retry backoff max delay milliseconds

gcs.retry.backoff.max.delay.ms

Maximum retry delay in milliseconds. The default value is 32000.

GCS retry backoff delay multiplier

gcs.retry.backoff.delay.multiplier

Retry delay multiplier. The default value is 2.0.

GCS retry backoff max attempts

gcs.retry.backoff.max.attempts

Retry max attempts. The default value is 6.

GCS retry backoff total timeout milliseconds

gcs.retry.backoff.total.timeout.ms

Retry total timeout in milliseconds. The default value is 50000.

Retry back-off

kafka.retry.backoff.ms

Retry backoff in milliseconds. In case of transient exceptions, useful for performing recovery. Maximum value is 86400000 (24 hours).

Error tolerance

errors.tolerance

Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records.

Dead letter queue topic name

errors.deadletterqueue.topic.name

The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ.

Dead letter queue topic replication factor

errors.deadletterqueue.topic .replication.factor

Replication factor used to create the dead letter queue topic when it doesn’t already exist.

Enable error context headers

errors.deadletterqueue.context .headers.enable

When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors.