# Flatten JSON Messages

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [labs-full.txt](https://docs.redpanda.com/labs-full.txt)

---
title: Flatten JSON Messages
latest-operator-version: v26.1.4
latest-console-tag: v3.7.3
latest-connect-version: 4.93.0
latest-redpanda-tag: v26.1.9
docname: flatten-go
page-component-name: labs
page-version: master
page-component-version: master
page-component-title: Labs
page-relative-src-path: flatten-go.adoc
page-edit-url: https://github.com/redpanda-data/redpanda-labs/edit/main/docs/modules/data-transforms/pages/flatten-go.adoc
description: Flatten JSON messages in topics using data transforms.
page-git-created-date: "2025-05-06"
page-git-modified-date: "2025-05-06"
---

<!-- Source: https://docs.redpanda.com/labs/data-transforms/flatten-go.md -->

This example uses Redpanda data transforms to take JSON messages in an input topic and flatten them using a customizable delimiter.

Example input topic

```json
{
  "content": {
    "id": 123,
    "name": {
      "first": "Dave",
      "middle": null,
      "last": "Voutila"
    },
    "data": [1, "fish", 2, "fish"]
  }
}
```

Example output topic with flattened JSON

```json
{
  "content.id": 123,
  "content.name.first": "Dave",
  "content.name.middle": null,
  "content.name.last": "Voutila",
  "content.data": [1, "fish", 2, "fish"]
}
```

## [](#prerequisites)Prerequisites

You must have the following:

-   At least version 1.20 of [Go](https://go.dev/doc/install) installed on your host machine.

-   [Install `rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) on your host machine.

-   [Docker and Docker Compose](https://docs.docker.com/compose/install/) installed on your host machine.


## [](#limitations)Limitations

-   Arrays of objects are currently untested.

-   Providing a series of objects as input, not an an array, may result in a series of flattened objects as output.

-   Due to how JSON treats floating point values, values such as `1.0` that can be converted to an integer will lose the decimal point. For example `1.0` becomes `1`.


## [](#run-the-lab)Run the lab

1.  Clone this repository:

    ```bash
    git clone https://github.com/redpanda-data/redpanda-labs.git
    ```

2.  Change into the `data-transforms/flatten/` directory:

    ```bash
    cd redpanda-labs/data-transforms/go/flatten
    ```

3.  Set the `REDPANDA_VERSION` environment variable to at least version v23.3.1. Data transforms was introduced in this version. For all available versions, see the [GitHub releases](https://github.com/redpanda-data/redpanda/releases).

    For example:

    ```bash
    export REDPANDA_VERSION=v26.1.9
    ```

4.  Set the `REDPANDA_CONSOLE_VERSION` environment variable to the version of Redpanda Console that you want to run. For all available versions, see the [GitHub releases](https://github.com/redpanda-data/redpanda/releases).

    > 📝 **NOTE**
    >
    > You must use at least version v3.0.0 of Redpanda Console to deploy this lab.

    For example:

    ```bash
    export REDPANDA_CONSOLE_VERSION=v3.7.3
    ```

5.  Start Redpanda in Docker by running the following command:

    ```bash
    docker compose up -d --wait
    ```

6.  Set up your rpk profile:

    ```bash
    rpk profile create flatten --from-profile profile.yml
    ```

7.  Create the required topics `iss_json` and `iss_avro`:

    ```bash
    rpk topic create src sink
    ```

8.  Build and deploy the transforms function:

    ```bash
    rpk transform build
    rpk transform deploy --input-topic=src --output-topic=sink
    ```

    This example accepts the following environment variables:

    -   `RP_FLATTEN_DELIM`: The delimiter to use when flattening the JSON fields. Defaults to `.`.

        For example:

        ```bash
        rpk transform deploy --var "RP_FLATTEN_DELIM=<delimiter>"
        ```


9.  Produce a JSON message to the source topic:

    ```bash
    rpk topic produce src
    ```

10.  Paste the following into the prompt and press Ctrl+C to exit:

     ```json
     {"message": "success", "timestamp": 1707743943, "iss_position": {"latitude": "-28.5723", "longitude": "-149.4612"}}
     ```

11.  Consume the sink topic to see the flattened result:

     ```bash
     rpk topic consume sink --num 1
     ```

     {
       "topic": "sink",
       "value": "{\\n  \\"message\\": \\"success\\"  \\"timestamp\\": 1.707743943e+09  \\"iss\_position.latitude\\": \\"-28.5723\\",\\n  \\"iss\_position.longitude\\": \\"-149.4612\\"\\n}\\n",
       "timestamp": 1707744765541,
       "partition": 0,
       "offset": 0
     }


You can also see this in [Redpanda Console](http://localhost:8080/topics/sink?p=-1&s=50&o=-1#messages).

## [](#clean-up)Clean up

To shut down and delete the containers along with all your cluster data:

```bash
docker compose down -v
```