rpk transform deploy

Deploy a data transform.

When run in the same directory as a transform.yaml, this reads the configuration file, then looks for a .wasm file with the same name as your project. If the input and output topics are specified in the configuration file, those are used. Otherwise, the topics can be specified on the command line using the --input-topic and --output-topic flags.

You can specify environment variables for the transform using the --var flag. Variables are separated by an equal sign. For example: --var=KEY=VALUE. The --var flag can be repeated to specify multiple variables.

You can specify the --from-offset flag to identify where on the input topic the transform should begin processing. Expressed as:

  • @T - Begin reading records with committed timestamp >= T (UNIX time, ms from epoch)

  • +N - Begin reading N records from the start of each input partition

  • -N - Begin reading N records prior to the end of each input partition

Note that the broker will only respect --from-offset on the first deploy for a given transform. Re-deploying the transform will cause processing to pick up at the last committed offset. This state is maintained until the transform is deleted.

Usage

rpk transform deploy [flags]
bash

Flags

Value Type Description

--compression

string

Output batch compression type.

--file

string

The WebAssembly module to deploy.

--from-offset

string

Process an input topic partition from a relative offset.

-h, --help

-

Help for deploy.

-i, --input-topic

string

The input topic to apply the transform to.

--name

string

The name of the transform.

-o, --output-topic

strings

The output topic to write the transform results to (repeatable).

--var

environmentVariable

Specify an environment variable in the form of KEY=VALUE.

--config

string

Redpanda or rpk config file; default search paths are /var/lib/redpanda/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml.

-X, --config-opt

stringArray

Override rpk configuration settings. See rpk -X or execute rpk -X help for inline detail or rpk -X list for terser detail.

--profile

string

Profile to use. See rpk profile for more details.

-v, --verbose

-

Enable verbose logging.

Examples

Deploy Wasm files directly without a transform.yaml file:

rpk transform deploy --file transform.wasm --name myTransform \
--input-topic my-topic-1 \
--output-topic my-topic-2
--output-topic my-topic-3
bash

Deploy a transformation with multiple environment variables:

rpk transform deploy --var FOO=BAR --var FIZZ=BUZZ
bash

Configure compression for batches output by data transforms. The default setting is none but you can choose from the following options:

  • none

  • gzip

  • snappy

  • lz4

  • zstd

Configure this at deployment using rpk with the --compression flag:

rpk transform deploy --compression <compression_type>
bash

Enabling compression may increase computation costs and could impact latency at the output topic.

For more details, see Sizing Guidelines.