23.2 23.1 22.3 22.2 22.1 21.11

Data Transforms API

Technical Preview

API reference documentation for Redpanda data transforms.

Technical preview features are not supported for production deployments.

Data transforms functions

OnRecordWritten

func OnRecordWritten(fn OnRecordWrittenCallback)

The OnRecordWritten function registers a callback of type OnRecordWrittenCallback, which is invoked when a record is written to the input topic.

The function should be called in a package’s main function to register the transform function that will be applied.

Data transforms types

OnRecordWrittenCallback

type OnRecordWrittenCallback func(e WriteEvent) ([]Record, error)

The OnRecordWrittenCallback type is a callback to transform records after a write event happens in the input topic. It’s the type of the parameter for the OnRecordWritten function.

Record

type Record struct {
	// Key is an optional field.
	Key []byte
	// Value is the blob of data that is written to Redpanda.
	Value []byte
	// Headers are client specified key/value pairs that are
	// attached to a record.
	Headers []RecordHeader
	// Attrs is the attributes of a record.
	//
	// Output records should leave these unset.
	Attrs RecordAttrs
	// The timestamp associated with this record.
	//
	// For output records this can be left unset as it will
	// always be the same value as the input record.
	Timestamp time.Time
	// The offset of this record in the partition.
	//
	// For output records this field is left unset,
	// as it will be set by Redpanda.
	Offset int64
}

The Record type is a record that has been written to Redpanda.

RecordAttrs

type RecordAttrs struct {
	// contains filtered or unexported fields
}

TimestampType

func (a RecordAttrs) TimestampType() int8

RecordHeader

type RecordHeader struct {
	Key   []byte
	Value []byte
}

The RecordHeader type is an optional key/value pair that is passed along with records.

WriteEvent

type WriteEvent interface {
	// Access the record associated with this event
	Record() Record
}

The WriteEvent type contains information about a record that was written.