Cloud
Also in Self-Managed

avro

Consume a stream of Avro OCF datum.

Avro JSON format

This scanner yields documents formatted as Avro JSON when decoding with Avro schemas. In this format the value of a union is encoded in JSON as follows:

  • if its type is null, then it is encoded as a JSON null;

  • otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.

For example, the union schema ["null","string","Foo"], where Foo is a record name, would encode:

  • null as null;

  • the string "a" as {"string": "a"}; and

  • a Foo instance as {"Foo": {…​}}, where {…​} indicates the JSON encoding of a Foo instance.

However, it is possible to instead create documents in standard/raw JSON format by setting the field raw_json to true.

Metadata

This scanner adds the following metadata fields to each message:

  • avro_schema: The Parsing Canonical Form of the Avro schema in the OCF file header.

  • avro_schema_fingerprint: The CRC-64-AVRO (Rabin) fingerprint of the schema’s canonical form, as an unsigned 64-bit integer.

  • Common

  • Advanced

scanner:
  avro: {}
scanner:
  avro:
    raw_json: false
    max_decompressed_block_bytes: 16777216

Fields

max_decompressed_block_bytes

The maximum size, in bytes, that a single compressed Avro OCF block is allowed to expand to when decompressed. This guards against decompression-amplification ("deflate bomb") inputs, where a tiny compressed block would otherwise expand into a very large allocation. A block that exceeds this limit causes the scan to fail rather than be materialized. There is deliberately no way to disable this cap: pipelines with legitimately larger blocks should set an explicit higher value. Set to 0 to use the default (equivalent to omitting the field).

Type: int

Default: 16777216

raw_json

Whether to decode messages into normal JSON rather than Avro JSON. When true, this unwraps union values (bare values instead of {"type": value} wrappers).

Type: bool

Default: false