Use Schema Registry in Redpanda Console

In Redpanda Console, the Schema Registry menu lists registered and verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to.

The Schema Registry is built into Redpanda, and you can use it with the Schema Registry API or with Redpanda Console. This section describes Schema Registry operations available in Redpanda Console.

Prerequisites

You must add a valid schemaRegistry configuration in Redpanda Console. For help configuring Redpanda Console to connect to Schema Registry, see Configure Message Deserialization in Redpanda Console.

Create or edit a schema

A schema is registered in the registry with a subject, which is a name that is associated with the schema as it evolves. To register a schema, click Create new schema.

  1. On the Create schema page, select the strategy type for how to derive the subject name.

    • Topic (default): The subject name is derived from the Redpanda topic name. See Topic strategy use case.

    • Record: The subject name is derived from the Kafka record name. See Record strategy use case.

    • TopicRecord: The subject name is derived from both topic name and record name, allowing for finer-grained schema organization. See TopicRecord strategy use case.

    • Custom: The subject name is user-defined.

  2. Select the serialization format with the schema definition.

  3. To build more complex schema definitions, add a reference to other schemas. For example, the two import statements are references to the PhoneNumber and Address schemas:

    {
syntax = "proto3";
import "PhoneNumber.proto";
import "Address.proto";
message Person {
    string name = 1;
    string email = 2;
    PhoneNumber phone  = 3;
    repeated Address address  = 4;
}

  4. After registering a schema, you can add a new version to it, change its compatibility, or delete it.

Topic strategy use case

The Topic strategy is suitable when you want to group schemas by the topics to which they are associated. Suppose you’re tracking product order information in a topic named Transactions. When a producer sends records to the OrderInfo topic, you want the record names to look something like:

  • Transactions - Record1

  • Transactions - Record2

Where Record1 and Record2 are unique identifiers. This is usually defined in your producer settings. Create your schema with the Topic strategy, and the subject name is always Transactions, with all customer transactions under the same topic.

Record strategy use case

The Record strategy is most useful when you have multiple schemas within a topic and need more granular categorization that’s influenced by the record name. Suppose there’s an Events topic with event types A and B. You may want each of those event types to have their own subject, their own schemas, and their own fully-qualified record names (for example, com.example.EventTypeA). If each event type has its own schema with the Record strategy, then when producers send these event types to the Events topic, their subjects are those record names:

  • com.example.EventTypeA

  • com.example.EventTypeB

The record names in the Events topic look like this:

  • Events-com.example.EventTypeA-Record1

  • Events-com.example.EventTypeB-Record1

  • Events-com.example.EventTypeA-Record2

  • Events-com.example.EventTypeB-Record2

TopicRecord strategy use case

The TopicRecord strategy is suitable when you want to organize schemas based on both topics and logical record types. Suppose there’s a microservices architecture where different services produce to the same topic: SharedEvents. Each microservice has a schema of its own for the shared events, but each schema uses the TopicRecord strategy. This results in the following subject names:

  • SharedEvents-com.example.MicroserviceAEvent

  • SharedEvents-com.example.MicroserviceBEvent

The record names look like this:

  • SharedEvents-com.example.MicroserviceAEvent-Record1

  • SharedEvents-com.example.MicroserviceBEvent-Record1

  • SharedEvents-com.example.MicroserviceAEvent-Record2

  • SharedEvents-com.example.MicroserviceBEvent-Record2

This allows for multiple schemas to govern the same shared events for different microservices, allowing granular organization.

Configure schema compatibility

Applications are often modeled around a specific business object structure. As applications change and the shape of their data changes, producer schemas and consumer schemas may no longer be compatible. You can decide how a consumer handles data from a producer that uses an older or newer schema, and reduce the chance of consumers hitting deserialization errors.

You can configure different types of schema compatibility, which are applied to a subject when a new schema is registered. The Schema Registry supports the following compatibility types:

  • BACKWARD (default) - Consumers using the new schema (for example, version 10) can read data from producers using the previous schema (for example, version 9).

  • BACKWARD_TRANSITIVE - Consumers using the new schema (for example, version 10) can read data from producers using all previous schemas (for example, versions 1-9).

  • FORWARD - Consumers using the previous schema (for example, version 9) can read data from producers using the new schema (for example, version 10).

  • FORWARD_TRANSITIVE - Consumers using any previous schema (for example, versions 1-9) can read data from producers using the new schema (for example, version 10).

  • FULL - A new schema and the previous schema (for example, versions 10 and 9) are both backward and forward compatible with each other.

  • FULL_TRANSITIVE - Each schema is both backward and forward compatible with all registered schemas.

  • NONE - No schema compatibility checks are done.

Compatibility uses and constraints

  • A consumer that wants to read a topic from the beginning (for example, an AI learning process) benefits from backward compatibility. It can process the whole topic using the latest schema. This allows producers to remove fields and add attributes.

  • A real-time consumer that doesn’t care about historical events but wants to keep up with the latest data (for example, a typical streaming application) benefits from forward compatibility. Even if producers change the schema, the consumer can carry on.

  • Full compatibility can process historical data and future data. This is the safest option, but it limits the changes that can be done. This only allows for the addition and removal of optional fields.

If you make changes that are not inherently backward-compatible, you may need to change compatibility settings or plan a transitional period, updating producers and consumers to use the new schema while the old one is still accepted.

Schema format Backward-compatible tasks Not backward-compatible tasks

Avro

Add fields with default values

Make fields nullable

Remove fields

Change data types of fields

Change enum values

Change field constraints

Change record of field names

Protobuf

Add fields

Remove fields

Remove required fields

Change data types of fields

JSON

Add optional properties

Relax constraints, for example:

  • Decrease a minimum value or increase a maximum value

  • Decrease minItems, minLength, or minProperties; increase maxItems, maxLength, maxProperties

  • Add more property types (for example, "type": "integer" to "type": ["integer", "string"])

  • Add more enum values

  • Reduce multipleOf by an integral factor

  • Relaxing additional properties if additionalProperties was not previously specified as false

  • Removing a uniqueItems property that was false

Remove properties

Add required properties

Change property names and types

Tighten or add constraints

Delete a schema

Select a schema to soft-delete a version of it or all schemas of its subject. Schemas cannot be deleted if any other schemas reference it. A soft-deleted schema can be recovered, but a permanently-deleted schema cannot be recovered. Redpanda does not recommend permanently deleting schemas in a production environment.

Suggested reading