Convert Timestamps using Rust

This lab uses data transforms with the Redpanda Schema Registry to convert keys or values with timestamps across various formats. Written in Rust, the example shows how to transform numeric epoch values in milliseconds to string-based formats and vice versa.

Prerequisites

You must have the following:

  • At least version 1.75 of Rust installed on your host machine.

  • The Wasm target for Rust installed. To install this target, run the following:

    rustup target add wasm32-wasi

  • Install rpk on your host machine.

  • Docker and Docker Compose installed on your host machine.

Run the lab

  1. Clone this repository:

    git clone https://github.com/redpanda-data/redpanda-labs.git

  2. Change into the data-transforms/ts-converter/ directory:

    cd redpanda-labs/data-transforms/rust/ts-converter

  3. Set the REDPANDA_VERSION environment variable to at least version 24.1.2. For all available versions, see the GitHub releases.

    For example:

    export REDPANDA_VERSION=24.3.2

  4. Set the REDPANDA_CONSOLE_VERSION environment variable to the version of Redpanda Console that you want to run. For all available versions, see the GitHub releases.

    For example:

    export REDPANDA_CONSOLE_VERSION=2.8.1

  5. Start Redpanda in Docker by running the following command:

    docker compose up -d --wait

  6. Set up your rpk profile:

    rpk profile create ts-converter --from-profile profile.yml

  7. Create the required input topic:

    rpk topic create src sink

  8. Create a source schema:

    echo '{"type": "long", "name": "epoch"}' | tee epoch.avsc
rpk registry schema create src-value --schema epoch.avsc

  9. Deploy the transforms function:

    rpk transform build
rpk transform deploy --file ts-converter.wasm -i src -o sink --var "TIMESTAMP_TARGET_TYPE=string[%+]"

    This example accepts the following environment variables:

    • TIMESTAMP_TARGET_TYPE (required): The output conversion time. Must be one of:

      • string[<pattern>]: where <pattern> is a valid chrono format string

      • unix[<precision>]: where <precision> is either seconds, milliseconds, microseconds, or nanoseconds

      • date: which truncates to just the date portion of the timestamp

      • time: which truncates to just the 24-hour time of the timestamp

    • TIMESTAMP_MODE: one of key, value, or a fielded version like value[my-field]

    • TIMESTAMP_STRING_FORMAT: if your input data is a string, provide the chrono format for parsing

  10. Produce an example epoch in milliseconds using rpk topic produce:

    echo "1704310686988" | rpk topic produce src --schema-id topic

  11. Consume the sink topic with rpk and using new schema to see the string-based timestamp:

    rpk topic consume sink --use-schema-registry=value -n 1 -o -1
    {
  "topic": "sink",
  "value": "\"2024-01-03T19:38:06.988+00:00\"",
  "timestamp": 1715890281087,
  "partition": 0,
  "offset": 0
}

You can also see this in Redpanda Console.

Clean up

To shut down and delete the containers along with all your cluster data:

docker compose down -v