# Filter Messages into a New Topic using a Regex

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [labs-full.txt](https://docs.redpanda.com/labs-full.txt)

---
title: Filter Messages into a New Topic using a Regex
latest-operator-version: v26.1.4
latest-console-tag: v3.7.3
latest-connect-version: 4.93.0
latest-redpanda-tag: v26.1.9
docname: regex-go
page-component-name: labs
page-version: master
page-component-version: master
page-component-title: Labs
page-relative-src-path: regex-go.adoc
page-edit-url: https://github.com/redpanda-data/redpanda-labs/edit/main/docs/modules/data-transforms/pages/regex-go.adoc
description: Filter messages from one topic into another using regular expressions (regex) and data transforms.
page-git-created-date: "2025-05-06"
page-git-modified-date: "2025-05-06"
---

<!-- Source: https://docs.redpanda.com/labs/data-transforms/regex-go.md -->

This is an example of how to filter messages from one topic into another using regular expressions (regex) and Redpanda data transforms. If a source topic contains a key or value that matches the regex, it will be produced to the sink topic.

Regexes are implemented using Go’s `regexp` library, which uses the same syntax as RE2. See the [RE2 wiki](https://github.com/google/re2/wiki/Syntax) for help with syntax.

The regex used in this example matches the typical email address pattern.

## [](#prerequisites)Prerequisites

You must have the following:

-   At least version 1.20 of [Go](https://go.dev/doc/install) installed on your host machine.

-   [Install `rpk`](https://docs.redpanda.com/current/get-started/rpk-install/) on your host machine.

-   [Docker and Docker Compose](https://docs.docker.com/compose/install/) installed on your host machine.


## [](#run-the-lab)Run the lab

1.  Clone this repository:

    ```bash
    git clone https://github.com/redpanda-data/redpanda-labs.git
    ```

2.  Change into the `data-transforms/go/regex/` directory:

    ```bash
    cd redpanda-labs/data-transforms/regex
    ```

3.  Set the `REDPANDA_VERSION` environment variable to at least version v23.3.1. Data transforms was introduced in this version. For all available versions, see the [GitHub releases](https://github.com/redpanda-data/redpanda/releases).

    For example:

    ```bash
    export REDPANDA_VERSION=v26.1.9
    ```

4.  Set the `REDPANDA_CONSOLE_VERSION` environment variable to the version of Redpanda Console that you want to run. For all available versions, see the [GitHub releases](https://github.com/redpanda-data/redpanda/releases).

    > 📝 **NOTE**
    >
    > You must use at least version v3.0.0 of Redpanda Console to deploy this lab.

    For example:

    ```bash
    export REDPANDA_CONSOLE_VERSION=v3.7.3
    ```

5.  Start Redpanda in Docker by running the following command:

    ```bash
    docker compose up -d --wait
    ```

6.  Set up your rpk profile:

    ```bash
    rpk profile create regex --from-profile profile.yml
    ```

7.  Create the required topics:

    ```bash
    rpk topic create src sink
    ```

8.  Build the transforms function:

    ```bash
    rpk transform build
    ```

9.  Deploy the transforms function:

    ```bash
    ./deploy-transform.sh
    ```

    See the file `deploy-transform.sh` to understand the regex used in the transform. Only input that matches the regular expression will be transformed.

    This example accepts the following environment variables:

    -   `PATTERN` (**required**): The regex to match against records. Here, the regex finds messages containing email addresses.

    -   `MATCH_VALUE`: By default, the regex matches record keys, but if set to `true`, the regex will match values.


10.  Run `rpk topic produce`:

     ```bash
     rpk topic produce src
     ```

11.  Paste the following into the prompt and press Ctrl+C to exit:

     ```json
     Hello, please contact us at help@example.com.
     Hello, please contact us at support.example.com.
     Hello, please contact us at help@example.edu.
     ```

12.  Consume the sink topic to see that input lines containing email addresses were extracted and produced to the sink topic:

     ```bash
     rpk topic consume sink --num 2
     ```

     {
       "topic": "sink",
       "value": "Hello, please contact us at help@example.com.",
       "timestamp": 1714525578013,
       "partition": 0,
       "offset": 0
     }
     {
       "topic": "sink",
       "value": "Hello, please contact us at help@example.edu.",
       "timestamp": 1714525579192,
       "partition": 0,
       "offset": 1
     }


> 📝 **NOTE**
>
> The second input line, `Hello, please contact us at support.example.com.`, is not in the sink topic because it did not match the regex that identifies valid email addresses.

You can also see the `sink` topic contents in [Redpanda Console](http://localhost:8080/topics/sink?p=-1&s=50&o=-1#messages).

Switch to the [`src` topic](http://localhost:8080/topics/src?p=-1&s=50&o=-1#messages) to see all of the events, including the one that does not match the regex and is not in the `sink` topic.

## [](#clean-up)Clean up

To shut down and delete the containers along with all your cluster data:

```bash
docker compose down -v
```