Connect
Also in Cloud

avro

Consume a stream of Avro OCF datum.

Introduced in version 4.25.0.

Avro JSON format

This scanner yields documents formatted as Avro JSON when decoding with Avro schemas. In this format the value of a union is encoded in JSON as follows:

  • if its type is null, then it is encoded as a JSON null;

  • otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.

For example, the union schema ["null","string","Foo"], where Foo is a record name, would encode:

  • null as null;

  • the string "a" as {"string": "a"}; and

  • a Foo instance as {"Foo": {…​}}, where {…​} indicates the JSON encoding of a Foo instance.

However, it is possible to instead create documents in standard/raw JSON format by setting the field raw_json to true.

Metadata

This scanner adds the following metadata fields to each message:

  • avro_schema: The Parsing Canonical Form of the Avro schema in the OCF file header.

  • avro_schema_fingerprint: The CRC-64-AVRO (Rabin) fingerprint of the schema’s canonical form, as an unsigned 64-bit integer.

  • Common

  • Advanced

scanner:
  avro: {}
scanner:
  avro:
    raw_json: false
    max_decompressed_block_bytes: 16777216

Fields

max_decompressed_block_bytes

The maximum size, in bytes, that a single compressed Avro OCF block is allowed to expand to when decompressed. This guards against decompression-amplification ("deflate bomb") inputs, where a tiny compressed block would otherwise expand into a very large allocation. A block that exceeds this limit causes the scan to fail rather than be materialized. There is deliberately no way to disable this cap: pipelines with legitimately larger blocks should set an explicit higher value. Set to 0 to use the default (equivalent to omitting the field).

Requires version 4.109.0 or later.

Type: int

Default: 16777216

raw_json

Whether to decode messages into normal JSON rather than Avro JSON. When true, this unwraps union values (bare values instead of {"type": value} wrappers).

Type: bool

Default: false