Docs Self-Managed Manage Schema Registry Use the Schema Registry API You are viewing the Self-Managed v24.3 beta documentation. We welcome your feedback at the Redpanda Community Slack #beta-feedback channel. To view the latest available version of the docs, see v24.2. Use the Schema Registry API Schemas provide human-readable documentation for an API. They verify that data conforms to an API, support the generation of serializers for data, and manage the compatibility of evolving APIs, allowing new versions of services to be rolled out independently. The Schema Registry is built into Redpanda, and you can use it with the API or Redpanda Console. This section describes operations available in the Schema Registry API. The Redpanda Schema Registry has API endpoints that allow you to perform the following tasks: Register schemas for a subject. When data formats are updated, a new version of the schema can be registered under the same subject, allowing for backward and forward compatibility. Retrieve schemas of specific versions. Retrieve a list of subjects. Retrieve a list of schema versions for a subject. Configure schema compatibility checking. Query supported serialization formats. Delete schemas from the registry. The following examples cover the basic functionality of the Redpanda Schema Registry based on an example Avro schema called sensor_sample. This schema contains fields that represent a measurement from a sensor for the value of the sensor topic, as defined below. { "type": "record", "name": "sensor_sample", "fields": [ { "name": "timestamp", "type": "long", "logicalType": "timestamp-millis" }, { "name": "identifier", "type": "string", "logicalType": "uuid" }, { "name": "value", "type": "long" } ] } Prerequisites To run the sample commands and code in each example, follow these steps to set up Redpanda and other tools: You need a running Redpanda cluster. If you don’t have one, you can follow the quickstart to deploy a self-managed cluster. These examples assume that the Schema Registry is available locally at http://localhost:8081. If the Schema Registry is hosted on a different address or port in your cluster, change the URLs in the examples. Download the jq utility. Install curl or Python. You can also use rpk to interact with the Schema Registry. The rpk registry set of commands call the different API endpoints as shown in the curl and Python examples. If using Python, install the Requests module, then create an interactive Python session: import requests import json def pretty(text): print(json.dumps(text, indent=2)) base_uri = "http://localhost:8081" Query supported schema formats To get the supported data serialization formats in the Schema Registry, make a GET request to the /schemas/types endpoint: Curl Python curl -s "http://localhost:8081/schemas/types" | jq . res = requests.get(f'{base_uri}/schemas/types').json() pretty(res) This returns the supported serialization formats: [ "JSON", "PROTOBUF", "AVRO" ] Register a schema A schema is registered in the registry with a subject, which is a name that is associated with the schema as it evolves. Subjects are typically in the form <topic-name>-key or <topic-name>-value. To register the sensor_sample schema, make a POST request to the /subjects/sensor-value/versions endpoint with the Content-Type application/vnd.schemaregistry.v1+json: rpk Curl Python rpk registry schema create sensor-value --schema ~/code/tmp/sensor_sample.avro curl -s \ -X POST \ "http://localhost:8081/subjects/sensor-value/versions" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"schema": "{\"type\":\"record\",\"name\":\"sensor_sample\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},{\"name\":\"identifier\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"value\",\"type\":\"long\"}]}"}' \ | jq sensor_schema = { "type": "record", "name": "sensor_sample", "fields": [ { "name": "timestamp", "type": "long", "logicalType": "timestamp-millis" }, { "name": "identifier", "type": "string", "logicalType": "uuid" }, { "name": "value", "type": "long" } ] } res = requests.post( url=f'{base_uri}/subjects/sensor-value/versions', data=json.dumps({ 'schema': json.dumps(sensor_schema) }), headers={'Content-Type': 'application/vnd.schemaregistry.v1+json'}).json() pretty(res) This returns the version id unique for the schema in the Redpanda cluster: rpk Curl SUBJECT VERSION ID TYPE sensor-value 1 1 AVRO { "id": 1 } When you register an evolved schema for an existing subject, the version id is incremented by 1. Retrieve a schema To retrieve a registered schema from the registry, make a GET request to the /schemas/ids/<id> endpoint: rpk Curl Python rpk registry schema get --id 1 curl -s \ "http://localhost:8081/schemas/ids/1" \ | jq . res = requests.get(f'{base_uri}/schemas/ids/1').json() pretty(res) The rpk output returns the subject and version, and the HTTP response returns the schema: rpk Curl SUBJECT VERSION ID TYPE sensor-value 1 1 AVRO { "schema": "{\"type\":\"record\",\"name\":\"sensor_sample\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},{\"name\":\"identifier\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"value\",\"type\":\"long\"}]}" } List registry subjects To list all registry subjects, make a GET request to the /subjects endpoint: rpk Curl Python rpk registry subject list --format json curl -s \ "http://localhost:8081/subjects" \ | jq . res = requests.get(f'{base_uri}/subjects').json() pretty(res) This returns the subject: [ "sensor-value" ] Retrieve schema versions of a subject To query the schema versions of a subject, make a GET request to the /subjects/<subject-name>/versions endpoint. For example, to get the schema versions of the sensor-value subject: Curl Python curl -s \ "http://localhost:8081/subjects/sensor-value/versions" \ | jq . res = requests.get(f'{base_uri}/subjects/sensor-value/versions').json() pretty(res) This returns the version ID: [ 1 ] Retrieve a schema of a subject To retrieve a schema associated with a subject, make a GET request to the /subjects/<subject-name>/versions/<version-id> endpoint: rpk Curl Python rpk registry schema get sensor-value --schema-version 1 curl -s \ "http://localhost:8081/subjects/sensor-value/versions/1" \ | jq . res = requests.get(f'{base_uri}/subjects/sensor-value/versions/1').json() pretty(res) The rpk output returns the subject, and for HTTP requests, its associated schema as well: rpk Curl SUBJECT VERSION ID TYPE sensor-value 1 1 AVRO { "subject": "sensor-value", "id": 1, "version": 1, "schema": "{\"type\":\"record\",\"name\":\"sensor_sample\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},{\"name\":\"identifier\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"value\",\"type\":\"long\"}]}" } To get the latest version, use latest as the version ID: rpk Curl Python rpk registry schema get sensor-value --schema-version latest curl -s \ "http://localhost:8081/subjects/sensor-value/versions/latest" \ | jq . res = requests.get(f'{base_uri}/subjects/sensor-value/versions/latest').json() pretty(res) To get only the schema, append /schema to the endpoint path: Curl Python curl -s \ "http://localhost:8081/subjects/sensor-value/versions/latest/schema" \ | jq . res = requests.get(f'{base_uri}/subjects/sensor-value/versions/latest/schema').json() pretty(res) { "type": "record", "name": "sensor_sample", "fields": [ { "name": "timestamp", "type": "long", "logicalType": "timestamp-millis" }, { "name": "identifier", "type": "string", "logicalType": "uuid" }, { "name": "value", "type": "long" } ] } Configure schema compatibility As applications change and their schemas evolve, you may find that producer schemas and consumer schemas are no longer compatible. You decide how you want a consumer to handle data from a producer that uses an older or newer schema. Applications are often modeled around a specific business object structure. As applications change and the shape of their data changes, producer schemas and consumer schemas may no longer be compatible. You can decide how a consumer handles data from a producer that uses an older or newer schema, and reduce the chance of consumers hitting deserialization errors. You can configure different types of schema compatibility, which are applied to a subject when a new schema is registered. The Schema Registry supports the following compatibility types: BACKWARD (default) - Consumers using the new schema (for example, version 10) can read data from producers using the previous schema (for example, version 9). BACKWARD_TRANSITIVE - Consumers using the new schema (for example, version 10) can read data from producers using all previous schemas (for example, versions 1-9). FORWARD - Consumers using the previous schema (for example, version 9) can read data from producers using the new schema (for example, version 10). FORWARD_TRANSITIVE - Consumers using any previous schema (for example, versions 1-9) can read data from producers using the new schema (for example, version 10). FULL - A new schema and the previous schema (for example, versions 10 and 9) are both backward and forward compatible with each other. FULL_TRANSITIVE - Each schema is both backward and forward compatible with all registered schemas. NONE - No schema compatibility checks are done. Compatibility uses and constraints A consumer that wants to read a topic from the beginning (for example, an AI learning process) benefits from backward compatibility. It can process the whole topic using the latest schema. This allows producers to remove fields and add attributes. A real-time consumer that doesn’t care about historical events but wants to keep up with the latest data (for example, a typical streaming application) benefits from forward compatibility. Even if producers change the schema, the consumer can carry on. Full compatibility can process historical data and future data. This is the safest option, but it limits the changes that can be done. This only allows for the addition and removal of optional fields. If you make changes that are not inherently backward-compatible, you may need to change compatibility settings or plan a transitional period, updating producers and consumers to use the new schema while the old one is still accepted. Schema format Backward-compatible tasks Not backward-compatible tasks Avro Add fields with default values Make fields nullable Remove fields Change data types of fields Change enum values Change field constraints Change record of field names Protobuf Add fields Remove fields Remove required fields Change data types of fields JSON Add optional properties Relax constraints, for example: Decrease a minimum value or increase a maximum value Decrease minItems, minLength, or minProperties; increase maxItems, maxLength, maxProperties Add more property types (for example, "type": "integer" to "type": ["integer", "string"]) Add more enum values Reduce multipleOf by an integral factor Relaxing additional properties if additionalProperties was not previously specified as false Removing a uniqueItems property that was false Remove properties Add required properties Change property names and types Tighten or add constraints To set the compatibility type for a subject, make a PUT request to /config/<subject-name> with the specific compatibility type: rpk Curl Python rpk registry compatibility-level set sensor-value --level BACKWARD curl -s \ -X PUT \ "http://localhost:8081/config/sensor-value" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"compatibility": "BACKWARD"}' \ | jq . res = requests.put( url=f'{base_uri}/config/sensor-value', data=json.dumps( {'compatibility': 'BACKWARD'} ), headers={'Content-Type': 'application/vnd.schemaregistry.v1+json'}).json() pretty(res) This returns the new compatibility type: rpk Curl SUBJECT LEVEL ERROR sensor-value BACKWARD { "compatibility": "BACKWARD" } If you POST an incompatible schema change, the request returns an error. For example, if you try to register a new schema with the value field’s type changed from long to int, and compatibility is set to BACKWARD, the request returns an error due to incompatibility: Curl Python curl -s \ -X POST \ "http://localhost:8081/subjects/sensor-value/versions" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"schema": "{\"type\":\"record\",\"name\":\"sensor_sample\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},{\"name\":\"identifier\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"value\",\"type\":\"int\"}]}"}' \ | jq sensor_schema["fields"][2]["type"] = "int" res = requests.post( url=f'{base_uri}/subjects/sensor-value/versions', data=json.dumps({ 'schema': json.dumps(sensor_schema) }), headers={'Content-Type': 'application/vnd.schemaregistry.v1+json'}).json() pretty(res) The request returns this error: { "error_code": 409, "message": "Schema being registered is incompatible with an earlier schema for subject \"{sensor-value}\"" } For an example of a compatible change, register a schema with the value field’s type changed from long to double: Curl Python curl -s \ -X POST \ "http://localhost:8081/subjects/sensor-value/versions" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{"schema": "{\"type\":\"record\",\"name\":\"sensor_sample\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},{\"name\":\"identifier\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"value\",\"type\":\"double\"}]}"}' \ | jq sensor_schema["fields"][2]["type"] = "double" res = requests.post( url=f'{base_uri}/subjects/sensor-value/versions', data=json.dumps({ 'schema': json.dumps(sensor_schema) }), headers={'Content-Type': 'application/vnd.schemaregistry.v1+json'}).json() pretty(res) A successful registration returns the schema’s id: { "id": 2 } Reference a schema To build more complex schema definitions, you can add a reference to other schemas. The following example registers a Protobuf schema in subject test-simple with a message name Simple. rpk Curl rpk registry schema create test-simple --schema simple.proto SUBJECT VERSION ID TYPE test-simple 1 2 PROTOBUF curl -X POST -H 'Content-type: application/vnd.schemaregistry.v1+json' http://127.0.0.1:8081/subjects/test-simple/versions -d '{"schema": "syntax = \"proto3\";\nmessage Simple {\n string id = 1;\n}","schemaType": "PROTOBUF"}' {"id":2} This schema is then referenced in a new schema in a different subject named import. rpk Curl # --references flag takes the format {name}:{subject}:{schema version} rpk registry schema create import --schema import_schema.proto --references simple:test-simple:2 SUBJECT VERSION ID TYPE import 1 3 PROTOBUF curl -X POST -H 'Content-type: application/vnd.schemaregistry.v1+json' http://127.0.0.1:8081/subjects/import/versions -d '{"schema": "syntax = \"proto3\";\nimport \"simple\";\nmessage Test3 {\n Simple id = 1;\n}","schemaType": "PROTOBUF", "references": [{"name": "simple", "subject": "test-simple", "version":1}]}' {"id":3} You cannot delete a schema when it is used as a reference. rpk Curl rpk registry schema delete test-simple --schema-version 1 One or more references exist to the schema {magic=1,keytype=SCHEMA,subject=test-simple,version=1} curl -X DELETE -H 'Content-type: application/vnd.schemaregistry.v1+json' http://127.0.0.1:8081/subjects/test-simple/versions/1 {"error_code":42206,"message":"One or more references exist to the schema {magic=1,keytype=SCHEMA,subject=test-simple,version=1}"} Call the /subjects/test-simple/versions/1/referencedby endpoint to see the schema IDs that reference version 1 for subject test-simple. rpk Curl rpk registry schema references test-simple --schema-version 1 SUBJECT VERSION ID TYPE import 1 3 PROTOBUF curl -H 'Content-type: application/vnd.schemaregistry.v1+json' http://127.0.0.1:8081/subjects/test-simple/versions/1/referencedby [3] Delete a schema The Schema Registry API provides DELETE endpoints for deleting a single schema or all schemas of a subject: /subjects/<subject>/versions/<version> /subjects/<subject> Schemas cannot be deleted if any other schemas reference it. A schema can be soft deleted (impermanently) or hard deleted (permanently), based on the boolean query parameter permanent. A soft deleted schema can be retrieved and re-registered. A hard deleted schema cannot be recovered. Soft delete a schema To soft delete a schema, make a DELETE request with the subject and version ID (where permanent=false is the default parameter value): rpk Curl Python rpk registry schema delete sensor-value --schema-version 1 curl -s \ -X DELETE \ "http://localhost:8081/subjects/sensor-value/versions/1" \ | jq . res = requests.delete(f'{base_uri}/subjects/sensor-value/versions/1').json() pretty(res) This returns the ID of the soft deleted schema: rpk Curl Successfully deleted schema. Subject: "sensor-value", version: "1" 1 Doing a soft delete for an already deleted schema returns an error: rpk Curl Subject 'sensor-value' Version 1 was soft deleted. Set permanent=true to delete permanently { "error_code": 40406, "message": "Subject 'sensor-value' Version 1 was soft deleted.Set permanent=true to delete permanently" } To list subjects of soft-deleted schemas, make a GET request with the deleted parameter set to true, /subjects?deleted=true: rpk Curl Python rpk registry subject list --deleted curl -s \ "http://localhost:8081/subjects?deleted=true" \ | jq . payload = { 'deleted' : 'true' } res = requests.get(f'{base_uri}/subjects', params=payload).json() pretty(res) This returns all subjects, including deleted ones: [ "sensor-value" ] To undo a soft deletion, first follow the steps to retrieve the schema, then register the schema. Hard delete a schema Redpanda doesn’t recommend hard (permanently) deleting schemas in a production system. The DELETE APIs are primarily used during the development phase, when schemas are being iterated and revised. To hard delete a schema, use the --permanent flag with the rpk registry schema delete command, or for curl or Python, make two DELETE requests with the second request setting the permanent parameter to true (/subjects/<subject>/versions/<version>?permanent=true): rpk Curl Python rpk registry schema delete sensor-value --schema-version 1 --permanent curl -s \ -X DELETE \ "http://localhost:8081/subjects/sensor-value/versions/1" \ | jq . curl -s \ -X DELETE \ "http://localhost:8081/subjects/sensor-value/versions/1?permanent=true" \ | jq . res = requests.delete(f'{base_uri}/subjects/sensor-value/versions/1').json() pretty(res) payload = { 'permanent' : 'true' } res = requests.delete(f'{base_uri}/subjects/sensor-value/versions/1', params=payload).json() pretty(res) Each request returns the version ID of the deleted schema: rpk Curl Successfully deleted schema. Subject: "sensor-value", version: "1" 1 1 A request for a hard-deleted schema returns an error: rpk Curl Subject 'sensor-value' not found. { "error_code": 40401, "message": "Subject 'sensor-value' not found." } Use READONLY mode for disaster recovery The /mode endpoint allows you to put Schema Registry in read-only or read-write mode. A read-only Schema Registry does not accept direct writes. An active production cluster can replicate schemas to a read-only Schema Registry to keep it in sync, for example using Redpanda’s Schema Migration tool. Users in the disaster recovery (DR) site cannot update schemas directly, so the DR cluster has an exact replica of the schemas in production. In a failover due to a disaster or outage, you can set Schema Registry to read-write mode, taking over for the failed cluster and ensuring availability. If authentication is enabled on Schema Registry, only superusers can change global and subject-level modes. Get global mode To query the global mode for Schema Registry: curl http://localhost:8081/mode Set global mode curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": <mode>}' http://localhost:8081/mode Replace the <mode> placeholder with the desired mode: READONLY READWRITE Get mode for a subject This request returns an error if there is no specific mode set for the subject: curl http://localhost:8081/mode/<subject> To retrieve the mode regardless of whether or not the subject has a specific mode, use the defaultToGlobal parameter: curl http://localhost:8081/mode/<subject>?defaultToGlobal=true This request returns the mode that is enforced. If the subject is set to a specific mode (to override the global mode), it will return the override mode, otherwise it returns the global mode. Set mode for a subject curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"mode": "READONLY"}' http://localhost:8081/mode/<subject> Suggested reading Redpanda Schema Registry rpk registry Schema Registry API Broker Configuration Properties (search for schema_registry) Monitor Schema Registry service-level metrics Configure broker properties for Schema Registry Deserialization Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Redpanda Schema Registry Server-Side Schema ID Validation