Docs Labs Transform JSON Messages into a New Topic using JQ This lab contains a reusable data transform using jaq a rust version of the the popular jq command line JSON processor. See the jq manual for more information on how to write a filter: https://jqlang.github.io/jq/manual/ Prerequisites You must have the following: At least version 1.75 of Rust installed on your host machine. The Wasm target for Rust installed. To install this target, run the following: rustup target add wasm32-wasi Install rpk on your host machine. Docker and Docker Compose installed on your host machine. Run the lab Clone this repository: git clone https://github.com/redpanda-data/redpanda-labs.git Change into the data-transforms/jq/ directory: cd redpanda-labs/data-transforms/rust/jq Set the REDPANDA_VERSION environment variable to at least version 23.3.1. Data transforms was introduced in this version. For all available versions, see the GitHub releases. For example: export REDPANDA_VERSION=24.2.8 Set the REDPANDA_CONSOLE_VERSION environment variable to the version of Redpanda Console that you want to run. For all available versions, see the GitHub releases. For example: export REDPANDA_CONSOLE_VERSION=2.7.2 Start Redpanda in Docker by running the following command: docker compose up -d --wait Set up your rpk profile: rpk profile create jq --from-profile profile.yml Create the required topics: rpk topic create src sink Deploy the transforms function: rpk transform build rpk transform deploy --var=FILTER='del(.email)' --input-topic=src --output-topic=sink This example accepts the following environment variable: FILTER (required): The jq expression that will run on each record’s value. Run rpk topic produce: rpk topic produce src Paste the following into the prompt and press Ctrl+D to exit: {"foo":42,"email":"help@example.com"} Consume the sink topic to see the email address was deleted and the record produced to the sink topic: rpk topic consume sink --num 1 { "topic": "sink", "value": "{\"foo\":42}", "timestamp": 1707749921393, "partition": 0, "offset": 0 } You can also see this in Redpanda Console. Clean up To shut down and delete the containers along with all your cluster data: docker compose down -v Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution