Converters and Serialization

Converters handle the serialization and deserialization of data between a Redpanda topic and a connector.

Each Redpanda message is a key and value record. Record key and value converters are configured separately with the Redpanda message key format and Redpanda message value format properties. Key and value converters can be different.

If an external system requires structured data (like BigQuery or a SQL database), then you must provide data with a schema. Use the Avro, Protobuf, or JSON converter with a schema.

ByteArray converter

The ByteArray converter is the most primitive and high-throughput converter. Schema is ignored. This is the default converter type for managed connectors.

To use the converter, select the ByteArray option as a key or value message format.

String converter

The String converter is a high-throughput converter. Schema is ignored. All data is converted to a string.

To use the converter, select the String option as a key or value message format.

JSON converter

The JSON converter supports a JSON schema embedded in the message, where each message contains a schema. It results in a bigger message size. The connector needs a message schema to check message format.

To use the converter, select the JSON option as a key or value message format.

Example JSON message with embedded schema:

{
  "schema": {
    "type": "struct",
    "fields": [
      {
        "type": "int64",
        "optional": false,
        "field": "person_id"
      },
      {
        "type": "string",
        "optional": false,
        "field": "name"
      }
    ]
  },
  "payload": {
    "person_id": 1,
    "name": "Redpanda"
  }
}

If you consume JSON data with no message schema, the schema check for the connector must be disabled with the Message key JSON contains schema or Message value JSON contains schema option.

Avro converter

The Avro converter requires a schema in Schema Registry.

See also: Redpanda Schema Registry

CloudEvents converter

The CloudEvents converter is specific to Debezium PostgreSQL and MySQL source connectors.

See also: CloudEvents Converter documentation

Protobuf converter Beta

The Protobuf converter requires a schema in Schema Registry. Currently the converter only supports sink connectors. It works for primitive types (numbers, boolean, string) and enum types.

To use the converter, select the Protobuf option as a key or value message format.

See also: Redpanda Schema Registry