Deserialization

In Redpanda, the messages exchanged between producers and consumers contain raw bytes. Schemas work as an agreed-upon format, like a contract, for producers and consumers to serialize and deserialize those messages. If a producer breaks this contract, consumers can fail.

Redpanda Cloud automatically tries to deserialize incoming messages and displays them in human-readable format. It tests different deserialization strategies until it finds one with no errors. If no deserialization attempts are successful, Redpanda Cloud renders the byte array in a hex viewer. Sometimes, the payload is displayed in hex bytes because it’s encrypted or because it uses a serializer that Redpanda Cloud cannot deserialize. When this happens, Redpanda Cloud displays troubleshooting information. You can also download the raw bytes of the message to feed it directly to your client deserializer or share it with a support team.

All deserialized messages are rendered as JSON objects and can be used as JavaScript objects in JavaScript filters (push filters).

Display messages in a specific format

Redpanda Cloud tries to automatically identify the correct deserialization type by decoding the message’s key, value, or header with all available deserialization methods. To display your messages in another format:

  1. Open your topic.

  2. Click the cog icon.

  3. Click Deserialization.

  4. Choose a new deserializer for either the keys or values in your messages.

Supported deserializers include:

  • Plain text

  • Kafka’s internal binary formats; for example, the __consumer_offsets topic

  • JSON

  • JSON with Schema Registry encoding

  • Smile

  • XML

  • Avro with Schema Registry encoding

  • Protobuf

  • Protobuf with Schema Registry encoding

  • Messagepack (for topics explicitly enabled to test MessagePack)

  • UTF-8 / strings

  • uint8, uint16, uint32, uint64

Suggested reading