Cloud
Also in Self-Managed

azure_data_lake_gen2

Sends message parts as files to an Azure Data Lake Gen2 filesystem. Each file is uploaded with the filename specified with the path field.

This output writes files to an Azure Data Lake Storage Gen2 file system.

To set a different path value (file name) for each file, use function interpolations, which are calculated per message of a batch.

Supports multiple authentication methods but only one of the following is required:

  • storage_connection_string

  • storage_account and storage_access_key

  • storage_account and storage_sas_token

  • storage_account to access via DefaultAzureCredential

If multiple are set then the storage_connection_string is given priority.

If the storage_connection_string does not contain the AccountName parameter, please specify it in the storage_account field.

Performance

This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field max_in_flight.

output:
  label: ""
  azure_data_lake_gen2:
    storage_account: ""
    storage_access_key: ""
    storage_connection_string: ""
    storage_sas_token: ""
    filesystem: "" # No default (required)
    path: ${!counter()}-${!timestamp_unix_nano()}.txt
    max_in_flight: 64

Fields

filesystem

The name of the data lake storage file system you want to upload messages to.

This field supports interpolation functions.

Type: string

# Examples:
filesystem: messages-${!timestamp("2006")}

max_in_flight

The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput.

Type: int

Default: 64

path

The path (file name) of each message to upload to the data lake storage file system.

This field supports interpolation functions.

Type: string

Default: ${!counter()}-${!timestamp_unix_nano()}.txt

# Examples:
path: ${!counter()}-${!timestamp_unix_nano()}.json

# ---

path: ${!meta("kafka_key")}.json

# ---

path: ${!json("doc.namespace")}/${!json("doc.id")}.json

storage_access_key

The access key for the storage account. Use this field along with storage_account for authentication. This field is ignored when the storage_connection_string field is populated.

Type: string

Default: ""

storage_account

The storage account to access. This field is ignored when the storage_connection_string field is populated, unless the connection string does not contain the AccountName parameter.

Type: string

Default: ""

storage_connection_string

The connection string for the storage account. This field is required if storage_account is not set.

If the storage_connection_string field does not contain the AccountName parameter value, specify it in the storage_account field.

Type: string

Default: ""

storage_sas_token

The SAS token for the storage account. Use this field along with storage_account for authentication. This field is ignored when either the storage_connection_string or storage_access_key fields are populated.

Type: string

Default: ""