Connect

gcp_cloud_storage

Type:

Available in: Cloud, Self-Managed

Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix.

Introduced in version 3.43.0.

  • Common

  • Advanced

inputs:
  label: ""
  gcp_cloud_storage:
    bucket: "" # No default (required)
    prefix: ""
    credentials_json: ""
    scanner:
      to_the_end: {}
inputs:
  label: ""
  gcp_cloud_storage:
    bucket: "" # No default (required)
    prefix: ""
    credentials_json: ""
    scanner:
      to_the_end: {}
    delete_objects: false

Metadata

This input adds the following metadata fields to each message:

- gcs_key
- gcs_bucket
- gcs_last_modified
- gcs_last_modified_unix
- gcs_content_type
- gcs_content_encoding
- All user defined metadata

You can access these metadata fields using function interpolation.

Credentials

By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in Google Cloud Platform.

Fields

bucket

The name of the bucket from which to download objects.

Type: string

credentials_json

Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see Google’s service account documentation.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

delete_objects

Whether to delete downloaded objects from the bucket once they are processed.

Type: bool

Default: false

prefix

Optional path prefix, if set only objects with the prefix are consumed.

Type: string

Default: ""

scanner

The scanner by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the csv scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once.

Requires version 4.25.0 or later.

Type: scanner

Default:

to_the_end: {}