24.1 beta 23.3 23.2 23.1

Use Schema Registry in Redpanda Console

In Redpanda Console, the Schema Registry tab lists registered and verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to.

The Schema Registry is built into Redpanda, and you can use it with the API or Redpanda Console. This section describes Schema Registry operations available in Redpanda Console. See also:

Prerequisites

Redpanda Console is automatically configured with new Redpanda Cloud deployments. However, if you’re running Redpanda Console with self-hosted Redpanda, you must add a valid schemaRegistry configuration in Redpanda Console:

redpanda-console-config.yaml
kafka:
  schemaRegistry:
    enabled: true
    urls: ["https://my-schema-registry.com"]
    username: console
    password: redacted # Or set using flags or env variable
  # To enable protobuf support
  protobuf:
    enabled: true
    schemaRegistry:
      enabled: true
      refreshInterval: 5m

Create or edit a schema

A schema is registered in the registry with a subject, which is a name that is associated with the schema as it evolves. To register a schema, click Create new schema.

  1. On the Create schema page, select the strategy type for how to derive the subject name.

    • Topic (default): The subject name is derived from the Redpanda topic name. See Topic strategy use case.

    • Record: The subject name is derived from the Kafka record name. See Record strategy use case.

    • TopicRecord: The subject name is derived from both topic name and record name, allowing for finer-grained schema organization. See TopicRecord strategy use case.

    • Custom: The subject name is user-defined.

  2. Select the serialization format (Avro or Protobuf) with the schema definition.

    See also: Deserialization in Redpanda Console.

  3. To build more complex schema definitions, add a reference to other schemas. For example, the two import statements are references to the PhoneNumber and Address schemas:

    {
syntax = "proto3";
import "PhoneNumber.proto";
import "Address.proto";
message Person {
    string name = 1;
    string email = 2;
    PhoneNumber phone  = 3;
    repeated Address address  = 4;
}

  4. After registering a schema, you can add a new version to it, change its compatibility, or delete it.

Topic strategy use case

The Topic strategy is suitable when you want to group schemas by the topics to which they are associated. Suppose you’re tracking product order information in a topic named Transactions. When a producer sends records to the OrderInfo topic, you want the record names to look something like:

  • Transactions - Record1

  • Transactions - Record2

Where Record1 and Record2 are unique identifiers. This is usually defined in your producer settings. Create your schema with the Topic strategy, and the subject name is always Transactions, with all customer transactions under the same topic.

Record strategy use case

The Record strategy is most useful when you have multiple schemas within a topic and need more granular categorization that’s influenced by the record name. Suppose there’s an Events topic with event types A and B. You may want each of those event types to have their own subject, their own schemas, and their own fully-qualified record names (for example, com.example.EventTypeA). If each event type has its own schema with the Record strategy, then when producers send these event types to the Events topic, their subjects are those record names:

  • com.example.EventTypeA

  • com.example.EventTypeB

The record names in the Events topic look like this:

  • Events-com.example.EventTypeA-Record1

  • Events-com.example.EventTypeB-Record1

  • Events-com.example.EventTypeA-Record2

  • Events-com.example.EventTypeB-Record2

TopicRecord strategy use case

The TopicRecord strategy is suitable when you want to organize schemas based on both topics and logical record types. Suppose there’s a microservices architecture where different services produce to the same topic: SharedEvents. Each microservice has a schema of its own for the shared events, but each schema uses the TopicRecord strategy. This results in the following subject names:

  • SharedEvents-com.example.MicroserviceAEvent

  • SharedEvents-com.example.MicroserviceBEvent

The record names look like this:

  • SharedEvents-com.example.MicroserviceAEvent-Record1

  • SharedEvents-com.example.MicroserviceBEvent-Record1

  • SharedEvents-com.example.MicroserviceAEvent-Record2

  • SharedEvents-com.example.MicroserviceBEvent-Record2

This allows for multiple schemas to govern the same shared events for different microservices, allowing granular organization.

Configure schema compatibility

Applications are often modeled around a specific business object structure. As applications change and the shape of their data changes, producer schemas and consumer schemas may no longer be compatible. You can decide how a consumer handles data from a producer that uses an older or newer schema, and reduce the chance of consumers hitting deserialization errors.

You can configure different types of schema compatibility, which are applied to a subject when a new schema is registered. The Schema Registry supports the following compatibility types:

  • BACKWARD (default) - Consumers using the new schema (for example, version 10) can read data from producers using the previous schema (for example, version 9).

  • BACKWARD_TRANSITIVE - Consumers using the new schema (for example, version 10) can read data from producers using all previous schemas (for example, versions 1-9).

  • FORWARD - Consumers using the previous schema (for example, version 9) can read data from producers using the new schema (for example, version 10).

  • FORWARD_TRANSITIVE - Consumers using any previous schema (for example, versions 1-9) can read data from producers using the new schema (for example, version 10).

  • FULL - A new schema and the previous schema (for example, versions 10 and 9) are both backward and forward compatible with each other.

  • FULL_TRANSITIVE - Each schema is both backward and forward compatible with all registered schemas.

  • NONE - No schema compatibility checks are done.

Compatibility uses and constraints

  • A consumer that wants to read a topic from the beginning (for example, an AI learning process) benefits from backward compatibility. It can process the whole topic using the latest schema. This allows producers to remove fields and add attributes.

  • A real-time consumer that doesn’t care about historical events but wants to keep up with the latest data (for example, a typical streaming application) benefits from forward compatibility. Even if producers change the schema, the consumer can carry on.

  • Full compatibility can process historical data and future data. This is the safest option, but it limits the changes that can be done. This only allows for the addition and removal of optional fields.

If you make changes that are not inherently backward-compatible, you may need to change compatibility settings or plan a transitional period, updating producers and consumers to use the new schema while the old one is still accepted.

Backward-compatible tasks Not backward-compatible tasks

Avro

Add fields with default values

Make fields nullable

Remove fields

Change data types of fields

Change enum values

Change field constraints

Change record of field names

Protobuf

Add fields

Remove fields

Remove required fields

Change data types of fields

Delete a schema

Select a schema to soft-delete a version of it or all schemas of its subject. Schemas cannot be deleted if any other schemas reference it. A soft-deleted schema can be recovered, but a permanently-deleted schema cannot be recovered. Redpanda does not recommend permanently deleting schemas in a production environment.

Suggested reading