Server-side Schema ID Validation
You can use server-side schema ID validation for clients using Confluent’s SerDes format that produce to Redpanda brokers. You can also configure Redpanda to inspect and reject records with schema IDs that aren’t valid according to the configured Subject Name strategy and registered with the Schema Registry.
This feature requires an Enterprise license. To upgrade, contact Redpanda sales. |
About schema ID validation
Records produced to a topic may use a serializer/deserializer client library, such as Confluent’s SerDes library, to encode their keys and values according to a schema in Manage.
When a client produces a record, the schema ID for the topic is encoded in the record’s payload header. The schema ID must be associated with a subject and a version in the Schema Registry. That subject is determined by the subject name strategy, which maps the topic and schema onto a subject.
A client may be misconfigured with either the wrong schema or the wrong subject name strategy, resulting in unexpected data on the topic. A produced record for an unregistered schema shouldn’t be stored by brokers or fetched by consumers. Yet, it may not be detected or dropped until after it’s been fetched and a consumer deserializes its mismatched schema ID.
Schema ID validation enables brokers (servers) to detect and drop records that were produced with an incorrectly configured subject name strategy, that don’t conform to the SerDes wire format, or encode an incorrect schema ID. With schema ID validation, records associated with unregistered schemas are detected and dropped earlier, by a broker rather than a consumer.
Schema ID validation doesn’t verify that a record’s payload is correctly encoded according to the associated schema. Schema ID validation only checks that the schema ID encoded in the record is registered in the Schema Registry. |
Configure schema ID validation
To use schema ID validation:
Enable schema ID validation
By default, server-side schema ID validation is disabled in Redpanda. To enable schema ID validation, change the enable_schema_id_validation
cluster property from its default value of none
to either redpanda
or compat
:
-
none
: Schema validation is disabled (no schema ID checks are done). Associated topic properties cannot be modified. -
redpanda
: Schema validation is enabled. Only Redpanda topic properties are accepted. -
compat
: Schema validation is enabled. Both Redpanda and compatible topic properties are accepted.
For example, use rpk
to set the value of enable_schema_id_validation
to redpanda
through the Admin API:
rpk cluster config set enable_schema_id_validation redpanda --api-urls=<admin-api-IP>:9644
Set subject name strategy per topic
The subject name strategies supported by Redpanda:
Subject Name Strategy | Subject Name Source | Subject Name Format (Key) | Subject Name Format (Value) |
---|---|---|---|
|
Topic name |
|
|
|
Fully-qualified record name |
|
|
|
Both topic name and fully-qualified record name |
|
|
When schema ID validation is enabled, Redpanda uses TopicNameStrategy
by default.
To customize the subject name strategy per topic, set the following client topic properties:
-
Set
redpanda.key.schema.id.validation
totrue
to enable key schema ID validation for the topic, and setredpanda.key.subject.name.strategy
to the desired subject name strategy for keys of the topic (default:TopicNameStrategy
). -
Set
redpanda.value.schema.id.validation
totrue
to enable value schema ID validation for the topic, and setredpanda.value.subject.name.strategy
to the desired subject name strategy for values of the topic (default:TopicNameStrategy
).
The
The If |
To support schema ID validation for compressed topics, a Redpanda broker decompresses each batch written to it so it can access the schema ID. |
Configuration examples
Create a topic with with RecordNameStrategy
:
rpk topic create topic_foo \
--topic-config redpanda.value.schema.id.validation=true \
--topic-config redpanda.value.subject.name.strategy=RecordNameStrategy \
-X brokers=<broker-addr>:9092
Alter a topic to RecordNameStrategy
:
rpk topic alter-config topic_foo \
--set redpanda.value.schema.id.validation=true \
--set redpanda.value.subject.name.strategy=RecordNameStrategy \
-X brokers=<broker-addr>:9092