# Configure Message Deserialization in Redpanda Console

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt)

---
title: Configure Message Deserialization in Redpanda Console
latest-redpanda-tag: v25.3.11
latest-console-tag: v3.7.3
latest-operator-version: v26.1.4
# EOL = End-of-Life (support lifecycle status)
page-is-nearing-eol: "false"
page-is-past-eol: "false"
page-eol-date: November 19, 2026
latest-connect-version: 4.93.0
docname: config/deserialization
page-component-name: streaming
page-version: "25.3"
page-component-version: "25.3"
page-component-title: Streaming
page-relative-src-path: config/deserialization.adoc
page-edit-url: https://github.com/redpanda-data/docs/edit/v/25.3/modules/console/pages/config/deserialization.adoc
description: Learn how to configure Redpanda Console to use Schema Registry, Protobuf files, and other deserialization methods to ensure your data is correctly interpreted and displayed.
page-git-created-date: "2024-09-11"
page-git-modified-date: "2026-05-26"
support-status: supported
---

<!-- Source: https://docs.redpanda.com/streaming/25.3/console/config/deserialization.md -->

Redpanda Console provides tools for deserializing and inspecting messages in Kafka topics. This topic explains how to configure Redpanda Console to use Schema Registry, Protobuf files, and other deserialization methods to ensure your data is correctly interpreted and displayed.

## [](#sr)Use Schema Registry

The Schema Registry allows Redpanda Console to dynamically retrieve schemas for deserializing Avro, Protobuf, and JSON messages. This setup is important to ensure that messages are correctly interpreted based on your schema definitions.

See [Configure access to the Schema Registry](https://docs.redpanda.com/streaming/25.3/console/config/connect-to-redpanda/#sr).

## [](#protobuf-configuration)Protobuf configuration

Redpanda Console supports several methods for providing Protobuf schemas, including the Schema Registry, local file system, and GitHub repositories.

> 📝 **NOTE**
>
> You don’t need to provide standard types, such as Google’s timestamp, in your schemas. These standard types are included by default.

Most Kafka clients that serialize Protobuf messages put the serialized byte array into a binary wrapper that contains meta information, like the schema ID or the used prototypes, so the application that deserializes the Kafka records must recognize the format. The deserialization process requires Redpanda Console to be aware of the used Protobuf files as well as a mapping of what prototype should be used for each topic. This information can either be sourced from the Schema Registry or it can be provided with additional configuration so the files can be pulled from the local file system or a GitHub repository.

### [](#sr-protobuf)Use Schema Registry for Protobuf

If you use a Schema Registry for Protobuf deserialization, Redpanda Console can automatically fetch and use the required schemas without the need for manual topic mappings or additional configuration.

When using Schema Registry for Protobuf, you must not configure `serde.protobuf`. Redpanda Console detects and uses the Protobuf schemas from the Schema Registry automatically.

If you configure `serde.protobuf`, it enables manual deserialization mode, which requires you to specify topic mappings and source providers. Without those, Redpanda Console fails to start due to validation errors.

When using Schema Registry for Protobuf, you do not need to provide specific topic mappings, as the schemas will be fetched dynamically.

### [](#topic-mapping)Topic mapping

If you’re not [using a Schema Registry for Protobuf deserialization](#sr-protobuf), you must manually provide mappings between Kafka topics and their corresponding Protobuf types. This is necessary to inform Redpanda Console of the correct types to use for deserialization.

Consider a Kafka topic called `address-v1` and a corresponding `address.proto` file with the following structure:

`address.proto`

```proto
syntax = "proto3";
package fake_models;

option go_package = "pkg/protobuf";

message Address {
  int32 version = 1;
  string id = 2;
  message Customer {
    string customer_id = 1;
    string customer_type = 2;
  }
}
```

To map this topic to the Protobuf schema, use the following configuration:

#### Standalone

```yaml
serde:
  protobuf:
    enabled: true
    mappings:
    - topicName: address-v1
      valueProtoType: fake_models.Address # The full Protobuf type name
      # keyProtoType: Not specified because the key is a plain string
```

#### Kubernetes embedded

When Redpanda Console is part of the Redpanda Helm chart or Operator:
##### Operator

`redpanda-cluster`.yaml

```yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Console
metadata:
  name: redpanda-console
spec:
  clusterRef:
    name: redpanda
  config:
    serde:
      protobuf:
        enabled: true
        mappings:
        - topicName: address-v1
          valueProtoType: fake_models.Address
          # keyProtoType: Not specified because the key is a plain string
```

##### Helm

`redpanda-values.yaml`

```yaml
console:
  enabled: true
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
          - topicName: address-v1
            valueProtoType: fake_models.Address # The full Protobuf type name
            # keyProtoType: Not specified because the key is a plain string
```

#### Kubernetes standalone

When using the standalone Redpanda Console Helm chart:

`console-values.yaml`

```yaml
config:
  serde:
    protobuf:
      enabled: true
      mappings:
      - topicName: address-v1
        valueProtoType: fake_models.Address # The full Protobuf type name
        # keyProtoType: Not specified because the key is a plain string
```

-   `serde.protobuf.enabled`: Set to `true` to enable Protobuf deserialization.

-   `serde.protobuf.mappings.topicName`: The name of the Kafka topic.

-   `serde.protobuf.mappings.valueProtoType`: The fully-qualified Protobuf type for the message value.

-   `serde.protobuf.mappings.keyProtoType`: Specify the key Protobuf type if the key is not a plain string.


### [](#local-file-system)Local file system

You can mount Protobuf files directly from your local file system. Redpanda Console will search the specified paths for Protobuf files and build a registry with all the available types.

Configuration example:

#### Standalone

```yaml
serde:
  protobuf:
    enabled: true
    mappings:
      - topicName: orders
        valueProtoType: fake_models.Order
        keyProtoType: fake_models.OrderKey
    fileSystem:
      enabled: true
      # How often to refresh the Protobuf files from the filesystem
      refreshInterval: 5m
      # Directories containing the Protobuf files
      paths:
        - /etc/protos
```

#### Kubernetes embedded

When using the Redpanda Operator or the Redpanda Helm chart, configure Protobuf deserialization through the cluster configuration:
##### Operator

```yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Console
metadata:
  name: redpanda-console
spec:
  clusterRef:
    name: redpanda
  config:
    serde:
      protobuf:
        enabled: true
        mappings:
          - topicName: orders
            valueProtoType: fake_models.Order
            keyProtoType: fake_models.OrderKey
        fileSystem:
          enabled: true
          refreshInterval: 5m
          paths:
            - /etc/protos
```

##### Helm

```yaml
console:
  enabled: true
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: orders
              valueProtoType: fake_models.Order
              keyProtoType: fake_models.OrderKey
          fileSystem:
            enabled: true
            refreshInterval: 5m
            paths:
              - /etc/protos
```

#### Kubernetes standalone

When using the standalone Redpanda Console Helm chart:

```yaml
config:
  serde:
    protobuf:
      enabled: true
      mappings:
        - topicName: orders
          valueProtoType: fake_models.Order
          keyProtoType: fake_models.OrderKey
      fileSystem:
        enabled: true
        refreshInterval: 5m
        paths:
          - /etc/protos
```

Apply with:

```bash
helm upgrade --install redpanda-console redpanda/console -f console-values.yaml
```

-   `serde.protobuf.enabled`: Set to `true` to enable Protobuf deserialization.

-   `serde.protobuf.fileSystem.paths`: Paths to directories where Protobuf files are stored.

-   `serde.protobuf.fileSystem.refreshInterval`: The frequency at which Redpanda Console checks for updates to these files.


### [](#github-repository)GitHub repository

If your Protobuf files are stored in a GitHub repository, Redpanda Console can fetch and use them directly. This is particularly useful if your organization maintains Protobuf definitions in version control.

Configuration example:

#### Standalone

```yaml
serde:
  protobuf:
    enabled: true
    mappings:
      - topicName: orders
        valueProtoType: fake_models.Order
        keyProtoType: fake_models.OrderKey
    git:
      enabled: true
      repository:
        url: https://github.com/myorg/kafka-proto-files.git
        branch: master
      # How often to pull the git repository to refresh the schema files
      refreshInterval: 10m
      # Where all .proto files are stored in the git repository
      paths:
        - ./
```

#### Kubernetes embedded

When using the Redpanda Operator or the Redpanda Helm chart, configure GitHub repository access through the cluster configuration:
##### Operator

```yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Console
metadata:
  name: redpanda-console
spec:
  clusterRef:
    name: redpanda
  config:
    serde:
      protobuf:
        enabled: true
        mappings:
          - topicName: orders
            valueProtoType: fake_models.Order
            keyProtoType: fake_models.OrderKey
        git:
          enabled: true
          repository:
            url: https://github.com/myorg/kafka-proto-files.git
            branch: master
          refreshInterval: 10m
          paths:
            - ./
```

##### Helm

```yaml
console:
  enabled: true
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: orders
              valueProtoType: fake_models.Order
              keyProtoType: fake_models.OrderKey
          git:
            enabled: true
            repository:
              url: https://github.com/myorg/kafka-proto-files.git
              branch: master
            refreshInterval: 10m
            paths:
              - ./
```

#### Kubernetes standalone

When using the standalone Redpanda Console Helm chart:

```yaml
config:
  serde:
    protobuf:
      enabled: true
      mappings:
        - topicName: orders
          valueProtoType: fake_models.Order
          keyProtoType: fake_models.OrderKey
      git:
        enabled: true
        repository:
          url: https://github.com/myorg/kafka-proto-files.git
          branch: master
        refreshInterval: 10m
        paths:
          - ./
```

Apply with:

```bash
helm upgrade --install redpanda-console redpanda/console -f console-values.yaml
```

-   `serde.protobuf.enabled`: Set to `true` to enable Protobuf deserialization.

-   `serde.protobuf.git.repository.url`: The URL of the GitHub repository containing your Protobuf files.

-   `serde.protobuf.git.basicAuth`: Basic authentication credentials, often an API token for private repositories.

-   `serde.protobuf.git.refreshInterval`: Frequency at which the repository is polled for updates.


#### [](#private-git-repositories)Private Git repositories

If Protobuf files are stored in a private GitHub repository, Redpanda Console must authenticate using one of the following methods:

-   A [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) (PAT) over HTTPS (basic auth)

-   An [SSH private key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh)


##### [](#authenticate-using-a-github-personal-access-token-pat)Authenticate using a GitHub Personal Access Token (PAT)

Use this method to authenticate to GitHub over HTTPS using a personal access token.

###### Standalone

1.  Set environment variables:

    ```bash
    SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME=token
    SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD=<github-pat>
    ```

2.  Configure Redpanda Console:

    ```yaml
    serde:
      protobuf:
        enabled: true
        mappings:
          - topicName: <kafka-topic-name>
            valueProtoType: <protobuf-message-type>
        git:
          enabled: true
          repository:
            url: https://github.com/<github-organization>/<repository-name>.git
            branch: <branch-name>
          refreshInterval: 10m
          paths:
            - ./
          basicAuth:
            enabled: true
    ```

###### Kubernetes embedded

1.  Create a secret:

    ```yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: protobuf-git-auth
      namespace: redpanda
    type: Opaque
    stringData:
      SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD: <github-pat>
    ```

2.  Reference the secret and set the username:
###### Operator

```yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Console
metadata:
  name: redpanda-console
spec:
  clusterRef:
    name: redpanda
  extraEnv:
    - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME
      value: token
  extraEnvFrom:
    - secretRef:
        name: protobuf-git-auth
  config:
    serde:
      protobuf:
        enabled: true
        mappings:
          - topicName: <kafka-topic-name>
            valueProtoType: <protobuf-message-type>
        git:
          enabled: true
          repository:
            url: https://github.com/<github-organization>/<repository-name>.git
            branch: <branch-name>
          refreshInterval: 10m
          paths:
            - ./
          basicAuth:
            enabled: true
```

###### Helm

```yaml
console:
  enabled: true
  extraEnv:
    - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME
      value: token
  extraEnvFrom:
    - secretRef:
        name: protobuf-git-auth
  console:
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: <kafka-topic-name>
              valueProtoType: <protobuf-message-type>
          git:
            enabled: true
            repository:
              url: https://github.com/<github-organization>/<repository-name>.git
              branch: <branch-name>
            refreshInterval: 10m
            paths:
              - ./
            basicAuth:
              enabled: true
```

###### Kubernetes standalone

1.  Create a secret:

    ```yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: protobuf-git-auth
      namespace: redpanda
    type: Opaque
    stringData:
      SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD: <github-pat>
    ```

2.  Update Helm values:

    ```yaml
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: <kafka-topic-name>
              valueProtoType: <protobuf-message-type>
          git:
            enabled: true
            repository:
              url: https://github.com/<github-organization>/<repository-name>.git
              branch: <branch-name>
            refreshInterval: 10m
            paths:
              - ./
            basicAuth:
              enabled: true

    extraEnv:
      - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME
        value: token

    extraEnvFrom:
      - secretRef:
          name: protobuf-git-auth
    ```

Replace the following values:

-   `<github-pat>`: A GitHub personal access token that has `repo` scope

-   `<kafka-topic-name>`: Kafka topic to be deserialized

-   `<protobuf-message-type>`: Fully qualified Protobuf message type (for example, `com.example.Order`)

-   `<github-organization>`: GitHub organization or user that owns the repository

-   `<repository-name>`: Name of the repository containing `.proto` files

-   `<branch-name>`: Git branch to clone (for example, `main`)


#### [](#authenticate-using-ssh)Authenticate using SSH

Use this method to authenticate with GitHub over SSH using a private key.

##### Standalone

1.  Save the SSH private key on the local filesystem (for example, `/etc/redpanda/ssh/id_rsa`).

2.  Set environment variables:

    ```bash
    SERDE_PROTOBUF_GIT_SSH_ENABLED=true
    SERDE_PROTOBUF_GIT_SSH_USERNAME=git
    SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH=/etc/redpanda/ssh/id_rsa
    SERDE_PROTOBUF_GIT_SSH_PASSPHRASE=<private-key-passphrase>
    ```

3.  Configure Redpanda Console:

    ```yaml
    serde:
      protobuf:
        enabled: true
        mappings:
          - topicName: <kafka-topic-name>
            valueProtoType: <protobuf-message-type>
        git:
          enabled: true
          repository:
            url: git@github.com:<github-organization>/<repository-name>.git
            branch: <branch-name>
          refreshInterval: 10m
          paths:
            - ./
          ssh:
            enabled: true
    ```

##### Kubernetes embedded

1.  Create a secret with the SSH key:

    ```yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: protobuf-git-ssh
      namespace: redpanda
    type: Opaque
    stringData:
      privateKey: |
        -----BEGIN OPENSSH PRIVATE KEY-----
        <ssh-private-key>
        -----END OPENSSH PRIVATE KEY-----
      passphrase: <private-key-passphrase>
    ```

2.  Mount the secret and configure environment variables:
###### Operator

```yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Console
metadata:
  name: redpanda-console
spec:
  clusterRef:
    name: redpanda
  extraVolumeMounts:
    - name: git-ssh
      mountPath: /etc/git-ssh
      readOnly: true
  extraVolumes:
    - name: git-ssh
      secret:
        secretName: protobuf-git-ssh
  extraEnv:
    - name: SERDE_PROTOBUF_GIT_SSH_ENABLED
      value: "true"
    - name: SERDE_PROTOBUF_GIT_SSH_USERNAME
      value: git
    - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH
      value: /etc/git-ssh/privateKey
    - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE
      value: <private-key-passphrase>
```

###### Helm

```yaml
console:
  enabled: true
  extraVolumeMounts:
    - name: git-ssh
      mountPath: /etc/git-ssh
      readOnly: true
  extraVolumes:
    - name: git-ssh
      secret:
        secretName: protobuf-git-ssh
  extraEnv:
    - name: SERDE_PROTOBUF_GIT_SSH_ENABLED
      value: "true"
    - name: SERDE_PROTOBUF_GIT_SSH_USERNAME
      value: git
    - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH
      value: /etc/git-ssh/privateKey
    - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE
      value: <private-key-passphrase>
```

##### Kubernetes standalone

1.  Create a secret:

    ```yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: protobuf-git-ssh
      namespace: redpanda
    type: Opaque
    stringData:
      privateKey: |
        -----BEGIN OPENSSH PRIVATE KEY-----
        <ssh-private-key>
        -----END OPENSSH PRIVATE KEY-----
      passphrase: <private-key-passphrase>
    ```

2.  Update Helm values:

    ```yaml
    config:
      serde:
        protobuf:
          enabled: true
          mappings:
            - topicName: <kafka-topic-name>
              valueProtoType: <protobuf-message-type>
          git:
            enabled: true
            repository:
              url: git@github.com:<github-organization>/<repository-name>.git
              branch: <branch-name>
            refreshInterval: 10m
            paths:
              - ./
            ssh:
              enabled: true

    extraVolumeMounts:
      - name: git-ssh
        mountPath: /etc/git-ssh
        readOnly: true

    extraVolumes:
      - name: git-ssh
        secret:
          secretName: protobuf-git-ssh

    extraEnv:
      - name: SERDE_PROTOBUF_GIT_SSH_ENABLED
        value: "true"
      - name: SERDE_PROTOBUF_GIT_SSH_USERNAME
        value: git
      - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH
        value: /etc/git-ssh/privateKey
      - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE
        value: <private-key-passphrase>
    ```

Replace the following values:

-   `<kafka-topic-name>`: Kafka topic to be deserialized

-   `<protobuf-message-type>`: Fully qualified Protobuf message type (for example, `com.example.Order`)

-   `<github-organization>`: GitHub organization or user that owns the repository

-   `<repository-name>`: Name of the repository containing `.proto` files

-   `<branch-name>`: Git branch to clone (for example, `main`)

-   `<ssh-private-key>`: SSH private key content used to authenticate to GitHub (must be base64-safe if stored in secrets)

-   `<private-key-passphrase>`: Passphrase used to decrypt the SSH private key, if applicable


## [](#messagepack-deserialization)MessagePack deserialization

If your data is serialized using MessagePack, Redpanda Console can be configured to deserialize it.

### Standalone

```yaml
serde:
  messagePack:
    enabled: true
    # Define which topics use MessagePack serialization
    # Regex to match all topics by default
    topicNames: ["/.*/"]
```

### Kubernetes embedded

#### Operator

```yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Console
metadata:
  name: redpanda-console
spec:
  clusterRef:
    name: redpanda
  config:
    serde:
      messagePack:
        enabled: true
        topicNames:
          - "/.*/"
```

#### Helm

```yaml
console:
  enabled: true
  console:
    config:
      serde:
        messagePack:
          enabled: true
          topicNames: ["/.*/"]
```

### Kubernetes standalone

When using the standalone Redpanda Console Helm chart:

```yaml
config:
  serde:
    messagePack:
      enabled: true
      topicNames: ["/.*/"]
```

Apply with:

```bash
helm upgrade --install redpanda-console redpanda/console -f console-values.yaml
```

-   `serde.messagePack.enabled`: Enables MessagePack deserialization.

-   `serde.messagePack.topicNames`: A list of topic name regex patterns that specify which topics use MessagePack serialization. The default pattern (`/.*/`) matches all topics.


## [](#best-practices)Best practices

-   Use Schema Registry when possible.

    Schema Registry simplifies schema management and ensures that all messages are serialized and deserialized consistently across your Kafka ecosystem.

-   Organize Protobuf files.

    Whether using a local file system or a GitHub repository, keep your Protobuf files organized and use consistent naming conventions to avoid confusion.

-   Monitor deserialization performance.

    Regularly check the performance impact of deserialization, especially when using complex Protobuf schemas or large numbers of messages. Adjust refresh intervals and schema caching as needed.

-   Secure access.

    Ensure that credentials for accessing the Schema Registry or GitHub repositories are securely managed and rotated regularly.


## [](#troubleshooting)Troubleshooting

If you encounter issues with deserialization:

-   Ensure that the Schema Registry URL and credentials are correctly configured and accessible.

-   Check your topic mappings and Protobuf type names for accuracy.

-   Review the Redpanda Console for insights into any errors occurring during deserialization.