protobuf

Available in: Self-Managed

Handles conversions between JSON documents and protobuf messages using reflection, which allows you to make conversions from or to the target .proto files.

For more information about JSON mapping of protobuf messages, see ProtoJSON Format and Examples.

# Configuration fields, showing default values
label: ""
protobuf:
  operator: "" # No default (required)
  message: "" # No default (required)
  discard_unknown: false
  use_proto_names: false
  import_paths: []
  use_enum_numbers: false

Performance considerations

Processing protobuf messages using reflection is less performant than using generated native code. For scenarios where performance is critical, consider using Redpanda Connect plugins.

Operators

to_json

Converts protobuf messages into a generic JSON structure, which makes it easier to manipulate the contents of the JSON document within Redpanda Connect.

from_json

Attempts to create a target protobuf message from a generic JSON structure.

Fields

bsr[]

Buf Schema Registry configuration. Either this field or import_paths must be populated. Note that this field is an array, and multiple BSR configurations can be provided.

Type: object

Default: []

bsr[].api_key

Buf Schema Registry server API key, can be left blank for a public registry.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Manage Secrets before adding it to your configuration.

Type: string

Default: ""

bsr[].module

Module to fetch from a Buf Schema Registry e.g. 'buf.build/exampleco/mymodule'.

Type: string

bsr[].url

Buf Schema Registry URL, leave blank to extract from module.

Type: string

Default: ""

bsr[].version

Version to retrieve from the Buf Schema Registry, leave blank for latest.

Type: string

Default: ""

discard_unknown

When set to true, the from_json operator discards fields that are unknown to the schema.

Type: bool

Default: false

import_paths[]

A list of directories that contain .proto files, including all definitions required for parsing the target message. If left empty, the current directory is used. This processor imports all .proto files listed within specified or default directories.

Type: array

Default: []

message

The fully-qualified name of the protobuf message to convert from or to JSON.

Type: string

operator

The operator to execute.

Type: string

Options: to_json, from_json

use_enum_numbers

When set to true, the to_json operator deserializes enumeration fields as their numerical values instead of their string names. For example, an enum field with a value of ENUM_VALUE_ONE is represented as 1 in the JSON output.

Type: bool

Default: false

use_proto_names

When set to true, the to_json operator deserializes fields exactly as named in schema file.

Type: bool

Default: false

Examples

JSON to Protobuf using Schema from Disk

If we have the following protobuf definition within a directory called testing/schema:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of JSON documents of the form:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "caleb@myspace.com"
}

We can convert the documents into protobuf messages with the following config:

pipeline:
  processors:
    - protobuf:
        operator: from_json
        message: testing.Person
        import_paths: [ testing/schema ]

Protobuf to JSON using Schema from Disk

If we have the following protobuf definition within a directory called testing/schema:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of protobuf messages of the type Person, we could convert them into JSON documents of the format:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "caleb@myspace.com"
}

With the following config:

pipeline:
  processors:
    - protobuf:
        operator: to_json
        message: testing.Person
        import_paths: [ testing/schema ]

JSON to Protobuf using Buf Schema Registry

If we have the following protobuf definition within a BSR module hosted at buf.build/exampleco/mymodule:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of JSON documents of the form:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "caleb@myspace.com"
}

We can convert the documents into protobuf messages with the following config:

pipeline:
  processors:
    - protobuf:
        operator: from_json
        message: testing.Person
        bsr:
          - module: buf.build/exampleco/mymodule
            api_key: xxx

Protobuf to JSON using Buf Schema Registry

If we have the following protobuf definition within a BSR module hosted at buf.build/exampleco/mymodule:

syntax = "proto3";
package testing;

import "google/protobuf/timestamp.proto";

message Person {
  string first_name = 1;
  string last_name = 2;
  string full_name = 3;
  int32 age = 4;
  int32 id = 5; // Unique ID number for this person.
  string email = 6;

  google.protobuf.Timestamp last_updated = 7;
}

And a stream of protobuf messages of the type Person, we could convert them into JSON documents of the format:

{
	"firstName": "caleb",
	"lastName": "quaye",
	"email": "caleb@myspace.com"
}

With the following config:

pipeline:
  processors:
    - protobuf:
        operator: to_json
        message: testing.Person
        bsr:
          - module: buf.build/exampleco/mymodule
            api_key: xxxx