23.3 Beta 23.2 23.1 22.3 22.2 22.1 21.11

Protobuf Deserialization

If you have one or more topics that contain Protobuf serialized messages, you can configure Redpanda Console to deserialize the binary content into JSON, which makes the message human-readable. The resulting message can also be used in the programmable push filters, like a JavaScript object.

Protobuf serialization is commonly used with Kafka clusters in two ways:

  • With the Schema Registry (schema meta information gets embedded as part of the record’s value)

  • To write Protobuf serialized content into Kafka topics without a binary or custom wrapper

This is a technical difference, because most Kafka clients that serialize Protobuf messages put the serialized byte array into a binary wrapper that contains meta information such as the schema ID or the used prototypes. Therefore, the application that deserializes the Kafka records has to recognize the format. Redpanda Console supports both formats. The deserialization process requires Redpanda Console to be aware of the used .proto files as well as a mapping of what prototype should be used for each topic. This information can either be sourced from the Schema Registry or provided with additional configuration so the files can be pulled from the local file system or a GitHub repository.

To support imports, all prototypes are first registered in a proto registry, so your imports can be resolved. You must also ensure that all imported prototypes are part of the repository. Standard types (such as Google’s timestamp type) are included by default, so you don’t need to add them.

Schema Registry deserialization

Messages that have been serialized using Confluent’s KafkaProtobufSerializer can only be deserialized if the Schema Registry is configured. Unlike other providers, the Schema Registry does not require you to set up mappings that define which topics use which prototypes. Instead, this information is inferred from the messages, and the Schema Registry finds the right prototype for deserialization.

To set up the Schema Registry in Redpanda Console with self-hosted Redpanda, see Use Schema Registry in the UI. The Protobuf deserializer uses the same schema registry client that is configured under kafka.schemaRegistry. A valid configuration looks like this:

kafka:
  schemaRegistry:
    enabled: true
    urls: ["https://my-schema-registry.com"]
    username: console
    password: redacted # Or set using flags or env variable
  protobuf:
    enabled: true
    schemaRegistry:
      enabled: true # This tells the proto service to consider the schema registry when deserializing messages
      refreshInterval: 5m # How often the compiled proto schemas in the cache should be updated

Local file system

You can provide all required proto files by mounting them from your file system. All files must use the .proto file extension. In the config.yaml file, configure Redpanda Console to search one or more paths for your provided proto files so it can build a registry with all types:

kafka:
  protobuf:
    enabled: true
    mappings:
      - topicName: orders
        valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too)
        keyProtoType: package.Type
    fileSystem:
      enabled: true
      refreshInterval: 5m # 5min is the default refresh interval
      paths:
        - /etc/protos

GitHub repository

To provide the proto files through a GitHub repository, first place the files in a directory. The directory doesn’t matter, because Redpanda Console searches for all Proto files up to a directory depth of five levels. To use multiple import paths, set the importPaths property. For example, if your repository includes third-party Protobuf types, such as Google’s types, that are in a different directory to your own types, set the importPaths property to the paths of both directories.

Enable git in the config.yaml file:

kafka:
  protobuf:
    enabled: true
    mappings: []
      - topicName: xy
        valueProtoType: fake_model.Order # You can specify the proto type for the record key and/or value (just one will work too)
        keyProtoType: package.Type
    # importPaths is a list of paths from which to import Proto files into Redpanda Console.
    # Paths are relative to the root directory.
    # The `git` configuration must be enabled to use this feature.
    importPaths: []
    git:
      enabled: true
      refreshInterval: 5m
      repository:
        url: https://github.com/redpanda-data/owlshop-protos.git
      basicAuth:
        enabled: true
        username: token # API token from basic auth
        password: redacted

Topic mapping configuration

If you don’t use the Schema Registry for Protobuf deserialization, you must provide a mapping configuration, so Redpanda Console is aware of what Proto types it should use for each Kafka topic. Assume you have a Kafka topic called address-v1 and the respective address.proto file in your GitHub repository, which looks like the following:

syntax = "proto3";
package fake_models;

option go_package = "pkg/protobuf";

message Address {
  int32 version = 1;
  string id = 2;
  message Customer {
    string customer_id = 1;
    string customer_type = 2;
  }
}

The required mapping configuration looks like the following:

kafka:
  protobuf:
    enabled: true
    mappings:
    - topicName: address-v1
        valueProtoType: fake_model.Address # The full prototype URL is required
        # keyProtoType: The key is a plain string in Kafka, hence we don't have a prototype for the record's key