Rust SDK for Data Transforms

Rust SDK reference documentation for Redpanda data transforms.

The redpanda_transform_sdk crate contains the core functions and types for transforming data within Redpanda.

Functions

The Rust SDK includes the following function to manage data transforms to an output topic.

on_record_written

The on_record_written() function registers a callback that transforms records after they have been written to an input topic. This function is executed when a record is written to the topic. The callback is executed after the record has been written and fsynced to disk and the producer has been acknowledged.

pub fn on_record_written<E, F>(cb: F) -> !
where
    E: Debug,
    F: Fn(WriteEvent<'_>, &mut dyn RecordWriter) -> Result<(), E>,

Callback parameters

Example:

This transform function copies the same data from an input topic to an output topic.:

use anyhow::Result;
use redpanda_transform_sdk::*;

fn main() {
    on_record_written(my_transform);
}

// This will be called for each record in the source topic. The output records returned will be written to the destination topic.
fn my_transform(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> {
    writer.write(event.record)?;
    Ok(())
}

on_record_written() should be called from the main() function, since the method blocks and runs forever unless explicitly stopped. Carry out any necessary pre-processing steps before calling this method.

If an error is returned, Redpanda prints the error and aborts processing in the VM. Redpanda will then retry processing with backoff.

Structs

A struct is a custom data type that allows you to define a structure composed of related data attributes.

BorrowedHeader

BorrowedHeader is a zero-copy BorrowedRecord header.

pub struct BorrowedHeader<'a> { /* private fields */ }

impl<'a> BorrowedHeader<'a> {
    /// Create a new header without any copies.
    pub fn new(key: &'a [u8], value: Option<&'a [u8]>) -> Self;

    /// Returns the header's key.
    pub fn key(&self) -> &[u8];

    /// Returns the header's value or `None` if there is no value.
    pub fn value(&self) -> Option<&[u8]>;

    /// Clones this header obtaining ownership of the clone.
    pub fn to_owned(&self) -> RecordHeader;
}

BorrowedRecord

BorrowedRecord is a zero-copy representation of a Record in Redpanda.

pub struct BorrowedRecord<'a> { /* private fields */ }

impl<'a> BorrowedRecord<'a> {
    /// Create a new record without any copies.
    pub fn new(key: Option<&'a [u8]>, value: Option<&'a [u8]>) -> Self;

    /// Create a new record without any copies.
    pub fn new_with_headers(
        key: Option<&'a [u8]>,
        value: Option<&'a [u8]>,
        headers: Vec<BorrowedHeader<'a>>,
    ) -> Self;

    /// Returns the record's key or `None` if there is no key.
    pub fn key(&self) -> Option<&'a [u8]>;

    /// Returns the record's value or `None` if there is no value.
    pub fn value(&self) -> Option<&'a [u8]>;

    /// Return the headers for this record.
    pub fn headers(&self) -> &[BorrowedHeader<'a>];
}

Record

Record is a record in Redpanda. It consists of a key-value pair of bytes, along with a collection of RecordHeader.

Records are generated as the result of any transforms that act upon a BorrowedRecord.

pub struct Record { /* private fields */ }

impl Record {
    /// Create a new empty record with no key, no value and no headers.
    pub fn empty() -> Self;

    /// Create a new record with the given key and value.
    pub fn new(key: Option<Vec<u8>>, value: Option<Vec<u8>>) -> Self;

    /// Create a new record with the given, key, value and headers.
    pub fn new_with_headers(
        key: Option<Vec<u8>>,
        value: Option<Vec<u8>>,
        headers: Vec<RecordHeader>,
    ) -> Self;

    /// Returns the record's key or `None` if there is no key.
    pub fn key(&self) -> Option<&[u8]>;

    /// Sets the key for this record.
    pub fn set_key(&mut self, k: Vec<u8>);

    /// Returns the record's value or `None` if there is no value.
    pub fn value(&self) -> Option<&[u8]>;

    /// Sets the value for this record.
    pub fn set_value(&mut self, v: Vec<u8>);

    /// Append a header to this record.
    pub fn add_header(&mut self, header: RecordHeader);

    /// Returns a collection of headers for this record.
    pub fn headers(&self) -> impl ExactSizeIterator<Item = BorrowedHeader>;
}

RecordHeader

A RecordHeader is a key-value pair attached to a Record. Headers are opaque to the broker and are purely a mechanism for the producer and consumers to pass information.

pub struct RecordHeader { /* private fields */ }

impl RecordHeader {
    /// Create a new `RecordHeader`.
    pub fn new(key: Vec<u8>, value: Option<Vec<u8>>) -> Self;

    /// Returns the header's key.
    pub fn key(&self) -> &[u8];

    /// Sets the key for this header.
    pub fn set_key(&mut self, k: Vec<u8>);

    /// Returns the header's value or `None` if there is no value.
    pub fn value(&self) -> Option<&[u8]>;

    /// Sets the value for this header.
    pub fn set_value(&mut self, v: Vec<u8>);
}

RecordWriter

RecordWriter is a struct that writes transformed records to the output topic.

pub struct RecordWriter<'a> { /* private fields */ }

impl<'a> RecordWriter<'a> {
    // Creates a new [`RecordWriter`] using the specified `sink`.
    pub fn new(sink: &'a mut dyn RecordSink) -> Self;

    /// Write a record to the default output topic, returning any errors.
    // The default output topic is the 0th output topic in the transform's configuration,
    // or as specified in the --output-topic flag when running rpk transform deploy
    pub fn write<'b>(&mut self, r: impl Into<BorrowedRecord<'b>>) -> Result<(), WriteError>;

    /// Write a record with the given options, returning any errors.
    pub fn write_with_options<'b>(
        &mut self,
        r: impl Into<BorrowedRecord<'b>>,
        opts: WriteOptions<'b>,
    ) -> Result<(), WriteError>;
}

WriteEvent

WriteEvent is an event generated after the broker completes a write. A WriteEvent is asynchronously triggered after the broker acknowledges the producer’s write request, and is then passed to on_record_written.

pub struct WriteEvent<'a> { /* private fields */ }

WriteOptions

WriteOptions allows you to customize a RecordWriter's write.

pub struct WriteOptions<'a> { /* private fields */ }

impl<'a> WriteOptions<'a> {
    /// Create a new options struct with the [`Record`]'s destination to an optional `topic`.
    ///
    /// If `topic` is `None`, the record will be written to the first
    /// output topic listed in the configuration.
    pub fn new(topic: Option<&'a str>) -> Self;

    /// Create a new options struct with the [`Record`]'s destination to `topic`.
    pub fn to_topic(topic: &'a str) -> Self;
}

WrittenRecord

A written Record within Redpanda. A WrittenRecord is handed to on_record_written event handlers as the record that Redpanda wrote. The record contains a key-value pair with some headers and the record’s timestamp.

pub struct WrittenRecord<'a> { /* private fields */ }

impl<'a> WrittenRecord<'a> {
    /// Create a new record without any copies.
    ///
    /// NOTE: This method is useful for tests to mock out custom events to your transform function.
    pub fn from_record(record: impl Into<BorrowedRecord<'a>>, timestamp: SystemTime) -> Self;

    /// Create a new record without any copies.
    ///
    /// NOTE: This method is useful for tests to mock out custom events to your transform function.
    pub fn new(key: Option<&'a [u8]>, value: Option<&'a [u8]>, timestamp: SystemTime) -> Self;

    /// Create a new record without any copies.
    ///
    /// NOTE: This method is useful for tests to mock out custom events to your transform function.
    pub fn new_with_headers(
        key: Option<&'a [u8]>,
        value: Option<&'a [u8]>,
        timestamp: SystemTime,
        headers: Vec<BorrowedHeader<'a>>,
    ) -> Self;

    /// Returns the record's key or `None` if there is no key.
    pub fn key(&self) -> Option<&'a [u8]>;

    /// Returns the record's value or `None` if there is no value.
    pub fn value(&self) -> Option<&'a [u8]>;

    /// Returns the record's timestamp.
    ///
    /// NOTE: Record timestamps in Redpanda have millisecond resolution.
    pub fn timestamp(&self) -> SystemTime;

    /// Return the headers for this record.
    pub fn headers(&self) -> &[BorrowedHeader<'a>];
}

Enums

An enum is a custom data type that is defined by enumerating its possible variants.

WriteError

A WriteError can occur when writing records to the output topic.

#[non_exhaustive]
pub enum WriteError {
    Unknown(i32),
}

Traits

A trait defines behavior that a type shares with other types.

RecordSink

RecordSink is a trait that can receive a stream of records and output them to a destination topic.

pub trait RecordSink {
    // Required method
    fn write(&mut self, r: BorrowedRecord<'_>, opts: WriteOptions<'_>) -> Result<(), WriteError>;
}