Use Schema Registry in Redpanda Console
In Redpanda Console, the Schema Registry tab lists registered and verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to.
The Schema Registry is built into Redpanda, and you can use it with the API or Redpanda Console. This section describes Schema Registry operations available in Redpanda Console. See also: |
Prerequisites
Redpanda Console is automatically configured with new Redpanda Cloud deployments. However, if you’re running Redpanda Console with self-hosted Redpanda, you must add a valid schemaRegistry
configuration in Redpanda Console:
kafka:
schemaRegistry:
enabled: true
urls: ["https://my-schema-registry.com"]
username: console
password: redacted # Or set using flags or env variable
# To enable protobuf support
protobuf:
enabled: true
schemaRegistry:
enabled: true
refreshInterval: 5m
Create or edit a schema
A schema is registered in the registry with a subject, which is a name that is associated with the schema as it evolves. To register a schema, click Create new schema.
-
On the Create schema page, select the strategy type for how to derive the subject name.
-
Topic (default): The subject name is derived from the Redpanda topic name. See Topic strategy use case.
-
Record: The subject name is derived from the Kafka record name. See Record strategy use case.
-
TopicRecord: The subject name is derived from both topic name and record name, allowing for finer-grained schema organization. See TopicRecord strategy use case.
-
Custom: The subject name is user-defined.
-
-
Select the serialization format (Avro or Protobuf) with the schema definition.
See also: Deserialization in Redpanda Console.
-
To build more complex schema definitions, add a reference to other schemas. For example, the two
import
statements are references to thePhoneNumber
andAddress
schemas:{ syntax = "proto3"; import "PhoneNumber.proto"; import "Address.proto"; message Person { string name = 1; string email = 2; PhoneNumber phone = 3; repeated Address address = 4; }
json -
After registering a schema, you can add a new version to it, change its compatibility, or delete it.
Topic strategy use case
The Topic strategy is suitable when you want to group schemas by the topics to which they are associated. Suppose you’re tracking product order information in a topic named Transactions
. When a producer sends records to the OrderInfo
topic, you want the record names to look something like:
-
Transactions - Record1
-
Transactions - Record2
Where Record1
and Record2
are unique identifiers. This is usually defined in your producer settings. Create your schema with the Topic strategy, and the subject name is always Transactions
, with all customer transactions under the same topic.
Record strategy use case
The Record strategy is most useful when you have multiple schemas within a topic and need more granular categorization that’s influenced by the record name. Suppose there’s an Events
topic with event types A and B. You may want each of those event types to have their own subject, their own schemas, and their own fully-qualified record names (for example, com.example.EventTypeA
). If each event type has its own schema with the Record strategy, then when producers send these event types to the Events
topic, their subjects are those record names:
-
com.example.EventTypeA
-
com.example.EventTypeB
The record names in the Events topic look like this:
-
Events-com.example.EventTypeA-Record1
-
Events-com.example.EventTypeB-Record1
-
Events-com.example.EventTypeA-Record2
-
Events-com.example.EventTypeB-Record2
TopicRecord strategy use case
The TopicRecord strategy is suitable when you want to organize schemas based on both topics and logical record types. Suppose there’s a microservices architecture where different services produce to the same topic: SharedEvents
. Each microservice has a schema of its own for the shared events, but each schema uses the TopicRecord strategy. This results in the following subject names:
-
SharedEvents-com.example.MicroserviceAEvent
-
SharedEvents-com.example.MicroserviceBEvent
The record names look like this:
-
SharedEvents-com.example.MicroserviceAEvent-Record1
-
SharedEvents-com.example.MicroserviceBEvent-Record1
-
SharedEvents-com.example.MicroserviceAEvent-Record2
-
SharedEvents-com.example.MicroserviceBEvent-Record2
This allows for multiple schemas to govern the same shared events for different microservices, allowing granular organization.
Configure schema compatibility
Applications are often modeled around a specific business object structure. As applications change and the shape of their data changes, producer schemas and consumer schemas may no longer be compatible. You can decide how a consumer handles data from a producer that uses an older or newer schema, and reduce the chance of consumers hitting deserialization errors.
You can configure different types of schema compatibility, which are applied to a subject when a new schema is registered. The Schema Registry supports the following compatibility types:
-
BACKWARD
(default) - Consumers using the new schema (for example, version 10) can read data from producers using the previous schema (for example, version 9). -
BACKWARD_TRANSITIVE
- Consumers using the new schema (for example, version 10) can read data from producers using all previous schemas (for example, versions 1-9). -
FORWARD
- Consumers using the previous schema (for example, version 9) can read data from producers using the new schema (for example, version 10). -
FORWARD_TRANSITIVE
- Consumers using any previous schema (for example, versions 1-9) can read data from producers using the new schema (for example, version 10). -
FULL
- A new schema and the previous schema (for example, versions 10 and 9) are both backward and forward compatible with each other. -
FULL_TRANSITIVE
- Each schema is both backward and forward compatible with all registered schemas. -
NONE
- No schema compatibility checks are done.
Compatibility uses and constraints
-
A consumer that wants to read a topic from the beginning (for example, an AI learning process) benefits from backward compatibility. It can process the whole topic using the latest schema. This allows producers to remove fields and add attributes.
-
A real-time consumer that doesn’t care about historical events but wants to keep up with the latest data (for example, a typical streaming application) benefits from forward compatibility. Even if producers change the schema, the consumer can carry on.
-
Full compatibility can process historical data and future data. This is the safest option, but it limits the changes that can be done. This only allows for the addition and removal of optional fields.
If you make changes that are not inherently backward-compatible, you may need to change compatibility settings or plan a transitional period, updating producers and consumers to use the new schema while the old one is still accepted.
Backward-compatible tasks | Not backward-compatible tasks | |
---|---|---|
Avro |
Add fields with default values Make fields nullable |
Remove fields Change data types of fields Change enum values Change field constraints Change record of field names |
Protobuf |
Add fields Remove fields |
Remove required fields Change data types of fields |
Delete a schema
Select a schema to soft-delete a version of it or all schemas of its subject. Schemas cannot be deleted if any other schemas reference it. A soft-deleted schema can be recovered, but a permanently-deleted schema cannot be recovered. Redpanda does not recommend permanently deleting schemas in a production environment.