Server-Side Schema ID Validation
You can use server-side schema ID validation for clients using Confluent’s SerDes format that produce to Redpanda brokers. You can also configure Redpanda to inspect and reject records with schema IDs that aren’t valid according to the configured Subject Name strategy and registered with the Schema Registry.
|
This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales. If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply. |
About schema ID validation
Records produced to a topic may use a serializer/deserializer client library, such as Confluent’s SerDes library, to encode their keys and values according to a schema.
When a client produces a record, the serializer writes a schema locator for the record’s key or value. The locator is either the integer schema ID or the schema globally unique identifier (GUID). The serializer writes it to a reserved record header or to the start of the payload. The schema ID or GUID must be associated with a subject and a version in the Schema Registry. See Schema ID location. That subject is determined by the subject name strategy, which maps the topic and schema onto a subject.
A client may be misconfigured with either the wrong schema or the wrong subject name strategy, resulting in unexpected data on the topic. A produced record for an unregistered schema shouldn’t be stored by brokers or fetched by consumers. Yet, it may not be detected or dropped until after it’s been fetched and a consumer deserializes its mismatched schema ID.
Schema ID validation enables brokers (servers) to detect and drop records that were produced with an incorrectly configured subject name strategy, that don’t conform to the SerDes wire format, or encode an incorrect schema ID. With schema ID validation, records associated with unregistered schemas are detected and dropped earlier, by a broker rather than a consumer.
| Schema ID validation doesn’t verify that a record’s payload is correctly encoded according to the associated schema. Schema ID validation only checks that the schema ID or GUID in the record is registered in the Schema Registry. |
Schema ID location
In Redpanda 26.3 and later, schema ID validation reads the schema locator from a reserved record header before it checks the payload. This matches the header-based format in Confluent’s SerDes, so producers can keep the payload free of the wire format prefix. Producers that write the locator at the start of the payload, in the wire format, continue to work without changes.
Locator forms
A schema locator takes one of two forms. The first byte, called the magic byte, identifies the form:
| Magic byte | Followed by | Identifies |
|---|---|---|
|
A 4-byte big-endian integer |
The schema by its integer schema ID. This is the classic wire format. |
|
A 16-byte GUID |
The schema by its GUID. A GUID identifies the schema content across subjects. |
For Protobuf schemas, the message indexes follow the locator in the same location, either in the header value or in the payload.
How Redpanda selects the locator
Redpanda selects the locator for the key and for the value separately:
-
The key uses only the
__key_schema_idheader, and the value uses only the__value_schema_idheader. One never stands in for the other. -
If a record carries the header more than once, Redpanda reads only the last occurrence.
-
A header with a non-null value takes precedence over the payload. If Redpanda can’t decode the locator in that header, it rejects the record and doesn’t fall back to the payload.
-
If the header is absent or has a null value, Redpanda reads the locator from the start of the payload.
Header-based validation works in both the redpanda and compat modes of enable_schema_id_validation.
Context resolution
Redpanda resolves integer schema IDs and GUIDs only in the topic’s Schema Registry context, which the redpanda.schema.registry.context topic property sets. When the property isn’t set, Redpanda uses the default context (.). Redpanda doesn’t search other contexts and doesn’t recognize the confluent.schema.validation.context.name topic property.
A GUID can map to several schema IDs. Redpanda accepts the record if any of those IDs with the same schema type is registered under the subject that the subject name strategy expects. Soft-deleted versions count. Redpanda checks the highest ID first. If none match, Redpanda rejects the record.
The context that redpanda.schema.registry.context names must exist. If it doesn’t, Redpanda rejects every record on the topic that needs validation.
|
Prior to Redpanda 26.3, validation always looked up integer schema IDs and subjects in the default context, even when redpanda.schema.registry.context was set. After you upgrade to 26.3, validation uses the topic’s context instead. A topic might set redpanda.schema.registry.context to a context other than the default while its schemas are registered only in the default context. In that case, Redpanda rejects records on that topic that need validation. The same integer ID can belong to a different schema in the topic’s context. In that case, Redpanda accepts the record only if that schema is registered under the subject that the subject name strategy expects.
During a rolling upgrade, rejections start as soon as an upgraded broker leads a partition of the topic. Before you upgrade, do one of the following for each topic that sets both schema ID validation and redpanda.schema.registry.context:
-
If the topic’s producers use the default context, remove
redpanda.schema.registry.contextfrom the topic. In Redpanda 26.2, only Iceberg reads this property. -
Copy the topic’s schemas into the topic’s context with the same schema IDs and subjects. To keep the IDs, set the context to
IMPORTmode and import the schemas. Registering the schemas normally assigns new IDs, so Redpanda still rejects records that carry the old IDs.
You can’t change redpanda.schema.registry.context on an Iceberg topic while Iceberg is enabled, so for an Iceberg topic, copy the schemas. You can instead disable Iceberg to change the property. After you re-enable Iceberg, Redpanda uses the new context to resolve any records it hasn’t translated yet.
Rejected records
When validation is enabled for the key or the value, a non-null key or value must carry a valid locator in its reserved header or at the start of the payload. Otherwise, Redpanda rejects the record and the whole batch that contains it, and the producer receives an INVALID_RECORD error.
A null value (a tombstone) skips value validation. A null key is always rejected when key validation is enabled.
Write schema GUIDs to headers from a Java client
To write the schema GUID to the __key_schema_id or __value_schema_id header instead of the payload, set the schema ID serializer in a Confluent Java producer to HeaderSchemaIdSerializer. For example, for record values:
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
value.schema.id.serializer=io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer
Consumers that use Confluent’s DualSchemaIdDeserializer, the default schema ID deserializer, read the locator from the header or the payload.
HeaderSchemaIdSerializer and DualSchemaIdDeserializer first ship in Confluent Platform 8.0.0. Non-Java Confluent clients also support header-based schema IDs, starting with confluent-kafka-python, confluent-kafka-go, and confluent-kafka-dotnet 2.10.1, and confluent-kafka-javascript 1.3.1.
Monitor validation by locator type
The redpanda_kafka_schema_id_validation_attempts_total metric counts validation checks. Redpanda checks the key and the value separately, so a record whose key and value are both validated counts twice. Records whose locator Redpanda can’t decode aren’t counted. The metric’s schema_id_type label is integer_id or guid, and its source label is header or payload. Use it to track how much of your traffic uses header-based or GUID locators.
Configure schema ID validation
To use schema ID validation:
Enable schema ID validation
By default, server-side schema ID validation is disabled in Redpanda. To enable schema ID validation, change the enable_schema_id_validation cluster property from its default value of none to either redpanda or compat:
-
none: Schema validation is disabled (no schema ID checks are done). Associated topic properties cannot be modified. -
redpanda: Schema validation is enabled. Only Redpanda topic properties are accepted. -
compat: Schema validation is enabled. Both Redpanda and compatible topic properties are accepted.
For example, use rpk to set the value of enable_schema_id_validation to redpanda through the Admin API:
rpk cluster config set enable_schema_id_validation redpanda -X admin.hosts=<admin-api-IP>:9644
Set subject name strategy per topic
The subject name strategies supported by Redpanda:
| Subject Name Strategy | Subject Name Source | Subject Name Format (Key) | Subject Name Format (Value) |
|---|---|---|---|
|
Topic name |
|
|
|
Fully-qualified record name |
|
|
|
Both topic name and fully-qualified record name |
|
|
When schema ID validation is enabled, Redpanda uses TopicNameStrategy by default.
To customize the subject name strategy per topic, set the following client topic properties:
-
Set
redpanda.key.schema.id.validationtotrueto enable key schema ID validation for the topic, and setredpanda.key.subject.name.strategyto the desired subject name strategy for keys of the topic (default:TopicNameStrategy). -
Set
redpanda.value.schema.id.validationtotrueto enable value schema ID validation for the topic, and setredpanda.value.subject.name.strategyto the desired subject name strategy for values of the topic (default:TopicNameStrategy).
|
The
|
The redpanda. and confluent. properties are compatible. Either or both can be set simultaneously.
If subject.name.strategy is prefixed with confluent., the available subject name strategies must be prefixed with io.confluent.kafka.serializers.subject.. For example, io.confluent.kafka.serializers.subject.TopicNameStrategy.
| To support schema ID validation for compressed topics, a Redpanda broker decompresses each batch written to it so it can access the schema ID. |
Configuration examples
Create a topic with with RecordNameStrategy:
rpk topic create topic_foo \
--topic-config redpanda.value.schema.id.validation=true \
--topic-config redpanda.value.subject.name.strategy=RecordNameStrategy \
-X brokers=<broker-addr>:9092
Alter a topic to RecordNameStrategy:
rpk topic alter-config topic_foo \
--set redpanda.value.schema.id.validation=true \
--set redpanda.value.subject.name.strategy=RecordNameStrategy \
-X brokers=<broker-addr>:9092