Converters and Serialization
Converters handle the serialization and deserialization of data between a Redpanda topic and a connector.
Each Redpanda message is a key and value record. Record key and value converters are configured separately with the Redpanda message key format
and Redpanda message value format
properties. Key and value converters can be different.
If an external system requires structured data (like BigQuery or a SQL database), then you must provide data with a schema. Use the Avro, Protobuf, or JSON converter with a schema. |
ByteArray converter
The ByteArray converter is the most primitive and high-throughput converter. Schema is ignored. This is the default converter type for managed connectors.
To use the converter, select the ByteArray
option as a key or value message format.
String converter
The String converter is a high-throughput converter. Schema is ignored. All data is converted to a string.
To use the converter, select the String
option as a key or value message format.
JSON converter
The JSON converter supports a JSON schema embedded in the message, where each message contains a schema. It results in a bigger message size. The connector needs a message schema to check message format.
To use the converter, select the JSON
option as a key or value message format.
Example JSON message with embedded schema:
{
"schema": {
"type": "struct",
"fields": [
{
"type": "int64",
"optional": false,
"field": "person_id"
},
{
"type": "string",
"optional": false,
"field": "name"
}
]
},
"payload": {
"person_id": 1,
"name": "Redpanda"
}
}
If you consume JSON data with no message schema, the schema check for the connector must be disabled with the Message key JSON contains schema
or Message value JSON contains schema
option.
Avro converter
The Avro converter requires a schema in Schema Registry.
See also: Redpanda Schema Registry
CloudEvents converter
The CloudEvents converter is specific to Debezium PostgreSQL and MySQL source connectors.
See also: CloudEvents Converter documentation
Protobuf converter
The Protobuf converter requires a schema in Schema Registry. Currently the converter only supports sink connectors. It works for primitive types (numbers, boolean, string) and enum types.
To use the converter, select the Protobuf
option as a key or value message format.
See also: Redpanda Schema Registry