Redpanda Self-Managed Quickstart

Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days.

Redpanda Self-Managed is a modern streaming platform, compatible with Kafka APIs, designed for speed, simplicity, and efficiency. In this quickstart, you:

Deploy a three-broker Redpanda cluster.
Explore streaming data in Redpanda Console.
Learn the basics of streaming with the Redpanda CLI (rpk).
Deploy a streaming pipeline with Redpanda Connect.
Learn how to extend your trial license for an additional 30 days.

This quickstart uses Docker to run Redpanda, which is only for development and testing purposes. For production deployments, refer to the Linux deployment guides or the Kubernetes deployment guides. To download the Redpanda binary, see GitHub.

Looking for a managed solution? You can also get started quickly with a hosted Redpanda cluster by signing up for Redpanda Cloud.

Enterprise features

The Enterprise Edition of Redpanda Self-Managed adds advanced capabilities to help scale, manage, optimize, and secure your cluster.

All new Redpanda clusters include a built-in 30-day Enterprise Edition license, so you can evaluate enterprise features. Some features highlighted in this quickstart require an enterprise license:

Redpanda Console Authentication: Securely control login access to Redpanda Console.
Audit Logging: Track and monitor all user actions in a Redpanda topic.
Continuous Data Balancing: Automatically balance data across brokers to optimize performance.
Tiered Storage: Lower storage costs by offloading log segments to object storage.

Your trial license also includes additional enterprise features. For more information, see Redpanda Licenses and Enterprise Features.

Prerequisites

You need Docker Compose. For installation instructions, see the Docker Compose documentation.

To check if you have Docker Compose installed, run:

docker compose version

bash

You should see the version of Docker Compose that’s installed on your local machine.

If you’re on Windows, use a compatible terminal that supports Unix-like commands. Or, consider using Windows Subsystem for Linux (WSL).

Deploy Redpanda Self-Managed

To download, extract, and start Redpanda Self-Managed in Docker, run:

mkdir redpanda-quickstart && cd redpanda-quickstart && \
curl -sSL https://docs.redpanda.com/redpanda-quickstart.tar.gz | tar xzf - && \
cd docker-compose && docker compose up -d

bash

1 Create and navigate to the redpanda-quickstart directory.

Download and extract the archive.

Explore the downloaded files

docker-compose.yml

name: redpanda-quickstart-multi-broker
networks:
  redpanda_network:
    driver: bridge
volumes:
  redpanda-0: null
  redpanda-1: null
  redpanda-2: null
  minio: null
services:
  ##################
  # Redpanda Brokers #
  ##################
  redpanda-0:
    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      # Address the broker advertises to clients that connect to the Kafka API.
      # Use the internal addresses to connect to the Redpanda brokers
      # from inside the same Docker network.
      # Use the external addresses to connect to the Redpanda brokers
      # from outside the Docker network.
      - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
      # Address the broker advertises to clients that connect to the HTTP Proxy.
      - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
      # Redpanda brokers use the RPC API to communicate with each other internally.
      - --rpc-addr redpanda-0:33145
      - --advertise-rpc-addr redpanda-0:33145
      # Mode dev-container uses well-known configuration properties for development in containers.
      - --mode dev-container
      # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
      - --smp 1
      - --default-log-level=info
    image: docker.redpanda.com/redpandadata/redpanda:v24.3.9
    container_name: redpanda-0
    # Sets the username and password of the bootstrap SCRAM superuser
    # See https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#bootstrap-a-user-account
    environment:
      RP_BOOTSTRAP_USER: "superuser:secretpassword"
    volumes:
      - redpanda-0:/var/lib/redpanda/data
      - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml
    networks:
      - redpanda_network
    ports:
      - 18081:18081
      - 18082:18082
      - 19092:19092
      - 19644:9644
    healthcheck:
      test: ["CMD", "rpk", "cluster", "info", "-X", "user=superuser", "-X", "pass=secretpassword"]
      interval: 10s
      timeout: 5s
      retries: 10
    depends_on:
      - minio
  redpanda-1:
    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092
      - --advertise-kafka-addr internal://redpanda-1:9092,external://localhost:29092
      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:28082
      - --advertise-pandaproxy-addr internal://redpanda-1:8082,external://localhost:28082
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:28081
      - --rpc-addr redpanda-1:33145
      - --advertise-rpc-addr redpanda-1:33145
      - --mode dev-container
      - --smp 1
      - --default-log-level=info
      - --seeds redpanda-0:33145
    image: docker.redpanda.com/redpandadata/redpanda:v24.3.9
    container_name: redpanda-1
    environment:
      RP_BOOTSTRAP_USER: "superuser:secretpassword"
    volumes:
      - redpanda-1:/var/lib/redpanda/data
      - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml
    networks:
      - redpanda_network
    ports:
      - 28081:28081
      - 28082:28082
      - 29092:29092
      - 29644:9644
    depends_on:
      - redpanda-0
      - minio
  redpanda-2:
    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:39092
      - --advertise-kafka-addr internal://redpanda-2:9092,external://localhost:39092
      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:38082
      - --advertise-pandaproxy-addr internal://redpanda-2:8082,external://localhost:38082
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:38081
      - --rpc-addr redpanda-2:33145
      - --advertise-rpc-addr redpanda-2:33145
      - --mode dev-container
      - --smp 1
      - --default-log-level=info
      - --seeds redpanda-0:33145
    image: docker.redpanda.com/redpandadata/redpanda:v24.3.9
    container_name: redpanda-2
    environment:
      RP_BOOTSTRAP_USER: "superuser:secretpassword"
    volumes:
      - redpanda-2:/var/lib/redpanda/data
      - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml
    networks:
      - redpanda_network
    ports:
      - 38081:38081
      - 38082:38082
      - 39092:39092
      - 39644:9644
    depends_on:
      - redpanda-0
      - minio
  ####################
  # Redpanda Console #
  ####################
  console:
    container_name: redpanda-console
    image: docker.redpanda.com/redpandadata/console:v2.8.5
    networks:
      - redpanda_network
    entrypoint: /bin/sh
    command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml && echo "$$CONSOLE_ROLEBINDINGS_CONFIG_FILE" > /tmp/role-bindings.yml && /app/console'
    volumes:
      - ./config:/tmp/config/
    environment:
      CONFIG_FILEPATH: ${CONFIG_FILEPATH:-/tmp/config.yml}
      CONSOLE_CONFIG_FILE: |
        # Configure a connection to the Redpanda cluster
        # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/
        kafka:
          brokers: ["redpanda-0:9092"]
          schemaRegistry:
            enabled: true
            urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"]
          sasl:
            enabled: true
            username: superuser
            password: secretpassword
            mechanism: SCRAM-SHA-256
        redpanda:
          adminApi:
            enabled: true
            urls: ["http://redpanda-0:9644","http://redpanda-1:9644","http://redpanda-2:9644"]
            username: superuser
            password: secretpassword
        console:
          # Configures Redpanda Console to fetch topic documentation from GitHub and display it in the UI.
          # See https://docs.redpanda.com/current/console/config/topic-documentation/
          topicDocumentation:
            enabled: true
            git:
              enabled: true
              repository:
                url: https://github.com/redpanda-data/docs
                branch: main
                baseDirectory: tests/docker-compose
        login:
          enabled: true
          jwtSecret: change-this-to-something-secret
          useSecureCookies: false
          # Enable plain login authentication (enterprise feature)
          # See https://docs.redpanda.com/current/console/config/security/plain/
          plain:
            enabled: true
            credentials:
            - username: "jane"
              password: "some-other-secret-password"
            - username: "john"
              password: "some-secret-password"
        enterprise:
          # Configures role-based access control to data and features in Redpanda Console.
          # See https://docs.redpanda.com/current/console/config/security/authorization/
          rbac:
            enabled: true
            roleBindingsFilepath: /tmp/role-bindings.yml
      CONSOLE_ROLEBINDINGS_CONFIG_FILE: |
        roleBindings:
        - metadata:
            name: Platform Ops
          subjects:
            - kind: user
              provider: Plain
              name: jane
          roleName: admin
        - metadata:
            name: Software Engineers
          subjects:
            - kind: user
              provider: Plain
              name: john
          roleName: editor
    ports:
      - 8080:8080
    depends_on:
      redpanda-0:
        condition: service_healthy
  ####################
  # Redpanda Connect #
  ####################
  connect:
    container_name: redpanda-connect
    image: docker.redpanda.com/redpandadata/connect
    networks:
      - redpanda_network
    entrypoint: /bin/sh
    depends_on:
      redpanda-0:
        condition: service_healthy
    command: -c 'echo "$$CONNECT_CFG_FILE" > /tmp/connect.yml; /redpanda-connect -c /tmp/connect.yml'
    environment:
      # This Redpanda Connect configuration creates fake data,
      # processes it, and writes the output to a set of topics.
      #
      # Input:
      #  - Uses Redpanda Connect's generate input to generate fake data.
      # See https://docs.redpanda.com/redpanda-connect/components/inputs/generate/
      # Pipeline:
      #  - Bloblang mapping to batch each input and map 1 message to 'logins'
      #    topic, and a random number (1-3) of messages to 'transaction' topic
      #  - Unarchive processor to parse the JSON array and extract each
      #    element into its own message.
      # See https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/
      # Output:
      #  - kafka_franz output to write the messages to the Redpanda brokers.
      # See https://docs.redpanda.com/redpanda-connect/components/outputs/kafka_franz/
      CONNECT_CFG_FILE: |
        input:
          generate:
            interval: 1s
            mapping: |
              let first_name = fake("first_name")
              let last_name  = fake("last_name")

              root.user_id    = counter()
              root.name       = $$first_name + " " + $$last_name
              root.email      = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase()
              root.ip         = fake("ipv4")
              root.login_time = now()
        pipeline:
          processors:
            - mapping: |
                root = range(0, random_int(min:2, max:4)).map_each(cust -> this)
            - unarchive:
                format: "json_array"
            - mapping: |
                if batch_index() == 0 {
                  meta topic = "logins"
                  root = this
                } else {
                  meta topic       = "transactions"
                  root.user_id     = this.user_id
                  root.email       = this.email
                  root.index       = batch_index() - 1
                  root.product_url = fake("url")
                  root.price       = fake("amount_with_currency")
                  root.timestamp   = now()
                }
        output:
          kafka_franz:
            seed_brokers: [ "redpanda-0:9092" ]
            topic: $${! metadata("topic") }
            sasl:
              - mechanism: SCRAM-SHA-256
                password: secretpassword
                username: superuser
  ####################
  # rpk container to create the edu-filtered-domains topic #
  # See https://docs.redpanda.com/current/reference/rpk/rpk-topic/rpk-topic-create/
  ####################
  createtopic:
    command:
      - topic
      - create
      - edu-filtered-domains
      - -X user=superuser
      - -X pass=secretpassword
      - -X brokers=redpanda-0:9092
    image: docker.redpanda.com/redpandadata/redpanda:v24.3.9
    networks:
      - redpanda_network
    depends_on:
      redpanda-0:
        condition: service_healthy
  ####################
  # rpk container to register the schema #
  # See https://docs.redpanda.com/current/manage/schema-reg/schema-reg-api/
  ####################
  registerschema:
    command:
      - registry
      - schema
      - create
      - transactions
      - --schema
      - /etc/redpanda/transactions-schema.json
      - -X user=superuser
      - -X pass=secretpassword
      - -X registry.hosts=redpanda-0:8081
    image: docker.redpanda.com/redpandadata/redpanda:v24.3.9
    # Mount the local directory that contains your schema to the container.
    volumes:
      - ./transactions-schema.json:/etc/redpanda/transactions-schema.json
    networks:
      - redpanda_network
    depends_on:
      redpanda-0:
        condition: service_healthy
  ####################
  # rpk container to deploy the pre-built data transform #
  # See https://docs.redpanda.com/current/develop/data-transforms/deploy/
  ####################
  deploytransform:
    command:
      - transform
      - deploy
      - --file=/etc/redpanda/regex.wasm
      - --name=regex
      - --input-topic=logins
      - --output-topic=edu-filtered-domains
      - --var=PATTERN="[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.edu"
      - --var=MATCH_VALUE=true
      - -X user=superuser
      - -X pass=secretpassword
      - -X admin.hosts=redpanda-0:9644
    image: docker.redpanda.com/redpandadata/redpanda:v24.3.9
    volumes:
      - ./transform/regex.wasm:/etc/redpanda/regex.wasm
    networks:
      - redpanda_network
    depends_on:
      redpanda-0:
        condition: service_healthy
      createtopic:
        condition: service_started
  ####################
  # MinIO for Tiered Storage #
  # See https://min.io/
  #
  # NOTE: MinIO is included in this quickstart for development and evaluation purposes only.
  # It is not supported for production deployments of Redpanda.
  #
  # For production environments, use one of the supported object storage providers:
  # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/requirements/#object-storage-providers-for-tiered-storage
  ####################
  minio:
    container_name: minio
    image: minio/minio
    command: server --console-address ":9001" /data
    ports:
      - 9000:9000
      - 9001:9001
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: redpandaTieredStorage7
      MINIO_SERVER_URL: "http://minio:9000"
      MINIO_REGION_NAME: local
      MINIO_DOMAIN: minio
    volumes:
      - minio:/data
    networks:
      redpanda_network:
        aliases:
          - redpanda.minio
  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      - redpanda_network
    environment:
      - AWS_ACCESS_KEY_ID=minio
      - AWS_SECRET_ACCESS_KEY=redpandaTieredStorage7
      - AWS_REGION=local
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 minio redpandaTieredStorage7) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc mb minio/redpanda;
      /usr/bin/mc policy set public minio/redpanda;
      tail -f /dev/null
      "

yaml

rpk-profile.yaml

# This file configures `rpk` to connect to a remote Redpanda cluster running in the same local network as `rpk`.

# Configuration for connecting to the Kafka API of the Redpanda cluster.
kafka_api:
  # SASL (Simple Authentication and Security Layer) settings for authentication.
  sasl:
    user: superuser # The username used for authentication
    password: secretpassword # The password associated with the username
    mechanism: scram-sha-256 # Authentication mechanism; SCRAM-SHA-256 provides secure password-based authentication
  # List of Kafka brokers in the Redpanda cluster.
  # These brokers ensure high availability and fault tolerance for Kafka-based communication.
  brokers:
    - 127.0.0.1:19092 # Broker 1: Accessible on localhost, port 19092
    - 127.0.0.1:29092 # Broker 2: Accessible on localhost, port 29092
    - 127.0.0.1:39092 # Broker 3: Accessible on localhost, port 39092

# Configuration for connecting to the Redpanda Admin API.
# The Admin API allows you to perform administrative tasks such as managing configurations, monitoring, and scaling.
admin_api:
   # List of Admin API endpoints for managing the cluster.
  addresses:
    - 127.0.0.1:19644 # Admin API for Broker 1: Accessible on localhost, port 19644
    - 127.0.0.1:29644 # Admin API for Broker 2: Accessible on localhost, port 29644
    - 127.0.0.1:39644 # Admin API for Broker 3: Accessible on localhost, port 39644

yaml

bootstrap.yml

# =================================================================
# This file defines initial cluster properties for a Redpanda cluster.
# Some of these settings are intended for quickstart development and evaluation
# and are not suitable for production environments.
#
# For more information on bootstrap files, see:
# https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#configure-a-bootstrap-file
# =================================================================

#
# Enable SASL authentication for the Kafka and Admin APIs.
# https://docs.redpanda.com/current/reference/properties/cluster-properties/#admin_api_require_auth
admin_api_require_auth: true
# At least one superuser is required to be able to create other SASL users
# https://docs.redpanda.com/current/reference/properties/cluster-properties/#superusers
superusers:
  - superuser
# https://docs.redpanda.com/current/reference/properties/cluster-properties/#enable_sasl
enable_sasl: true
# Allow topics to be created on first access.
# https://docs.redpanda.com/current/reference/properties/cluster-properties/#auto_create_topics_enabled
auto_create_topics_enabled: true
# Enable data transforms.
# https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/
data_transforms_enabled: true
# Enable audit logging (enterprise feature).
# https://docs.redpanda.com/current/manage/audit-logging/
audit_enabled: true
# Enable Tiered Storage (enterprise feature).
# https://docs.redpanda.com/current/manage/tiered-storage/
cloud_storage_enabled: true
cloud_storage_region: local
cloud_storage_access_key: minio
cloud_storage_secret_key: redpandaTieredStorage7
cloud_storage_api_endpoint: minio
cloud_storage_api_endpoint_port: 9000
cloud_storage_disable_tls: true
cloud_storage_bucket: redpanda
# Forces segments to be uploaded to Tiered Storage faster for the purposes of the quickstart
# https://docs.redpanda.com/current/reference/properties/object-storage-properties/#cloud_storage_segment_max_upload_interval_sec
cloud_storage_segment_max_upload_interval_sec: 60
# Continuous Data Balancing (enterprise feature) continuously monitors your node and rack availability and disk usage. This enables self-healing clusters that dynamically balance partitions, ensuring smooth operations and optimal cluster performance.
# https://docs.redpanda.com/current/manage/cluster-maintenance/continuous-data-balancing/
partition_autobalancing_mode: continuous

yaml

3 Start Docker Compose.

When the containers are running, you have:

A Redpanda cluster with a 30-day free trial license.
A running data streaming pipeline generating data into the logins and transactions topics.
A Wasm data transform processing data from logins into edu-filtered-domains.
A secured instance of Redpanda Console with login authentication.

Explore Redpanda Console

Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. This section provides practical examples and scenarios to help you understand how to leverage Redpanda Console for different use cases, including data observability, Redpanda management, access control, and connectivity.

Redpanda Console was deployed as part of the docker-compose.yml file and is running locally on port 8080.

Log in to Redpanda Console

Redpanda Console is configured with plain login credentials (enterprise feature) for an admin user called jane. To start using Redpanda Console, you first need to log in with the credentials defined in the docker-compose.yml file.

Open your web browser and go to http://localhost:8080/login.
Enter the following credentials:
- Username: jane
- Password: some-other-secret-password
This user is an admin user that has access to all features, including the Admin panel.

You should now see an overview of your cluster’s status, health, and brokers.

To view details about a specific broker, click View at the end of the row in the Broker Details table.

View topics and filter messages

This quickstart deployment comes with two pre-configured topics, logins and transactions, actively receiving data from a Redpanda Connect pipeline. These topics simulate real-world data flows, enabling you to explore and interact with Redpanda Console features.

You can filter messages in topics using several methods:

Text search: Search for specific text within the message.
Partition offset filters: Set the start and end offsets to narrow down the message range.
JavaScript filters: Use JavaScript function bodies to create complex filters that run server-side, allowing you to filter through topics with millions of records.

Suppose you’re asked to find all transactions related to the .edu domain. You can use a JavaScript filter to display only messages that include email addresses in that domain.

In the left menu, click Topics.
Click the transactions topic. When the topic’s page opens, the Messages tab is selected by default.
Click Add filter > JavaScript Filter.
Give your filter a name, such as "Find .edu domains".
Replace the default JavaScript code with the following:
```
return value.email.includes(".edu");
```
js
Click Save to apply the filter.

You should see only messages that include the specific domain.

By default, the filter runs on the newest 50 messages. You can run the filter on older messages by changing the start offset. For example, to start from the oldest messages, select Beginning.

Explore Schema Registry

On the Schema Registry page, you see an overview of your schemas. You can create, manage, and inspect your schemas without leaving Redpanda Console.

Configure access control for Redpanda

Managing access control lists (ACLs) can be complex, but Redpanda Console simplifies this with an intuitive interface.

This quickstart deployment includes a bootstrapped superuser with full administrative privileges. The superuser has unrestricted access to all topics, consumer groups, transactions, and cluster-level operations. By default, the credentials for the superuser are:

Username: superuser
Password: secretpassword

For improved security and role-based access control (RBAC), you should create additional users with restricted permissions as needed.

On the Security page, you can:

Create, view, and edit ACLs for topics, consumer groups, transactions, and the entire cluster.
Quickly understand what each principal (user) is authorized to do.

Suppose you’re onboarding a new team member who needs access to specific topics. You can use Redpanda Console to ensure users have the right permissions to perform their tasks.

In the left menu, click Security.
On the Users tab, click Create user.
Enter "Sasha" in the username field.
Enter "sashapassword" in the password field.
Click Create.

Click Done, and you see a new user called Sasha. This user has no permissions yet. To set permissions on the transactions topic:

On the Access control page, open to the Roles tab.
Click Create role.
Enter "transaction-managers" as the role name.
In the topic selector dropdown, select Literal and enter "transactions" in the input field.
Under Operations, click the All dropdown and select Allow.
Scroll down to the bottom of the page and under Principals select Sasha from the dropdown.
Click Create.

Now Sasha has full access only to the topic called transactions.

To test these permissions, use rpk to connect to Redpanda as the user Sasha.

Try to access the logins topic as Sasha:

docker exec -it redpanda-0 rpk topic describe logins \
  -X user=Sasha \
  -X pass=sashapassword \
  -X sasl.mechanism=SCRAM-SHA-256 \
  -X brokers=redpanda-0:9092

bash

You are not authorized to view the topic.

SUMMARY
=======
NAME        logins
PARTITIONS  0
ERROR       TOPIC_AUTHORIZATION_FAILED: Not authorized to access topics: [Topic authorization failed.]

CONFIGS
=======
config response contained error: TOPIC_AUTHORIZATION_FAILED: Not authorized to access topics: [Topic authorization failed.]

Now try to access the transactions topic as Sasha:

docker exec -it redpanda-0 rpk topic describe transactions \
  -X user=Sasha \
  -X pass=sashapassword \
  -X sasl.mechanism=SCRAM-SHA-256 \
  -X brokers=redpanda-0:9092

bash

You have access to this topic.

Example output

SUMMARY
=======
NAME        transactions
PARTITIONS  1
REPLICAS    1

CONFIGS
=======
KEY                                   VALUE       SOURCE
cleanup.policy                        delete      DEFAULT_CONFIG
compression.type                      producer    DEFAULT_CONFIG
delete.retention.ms                   -1          DEFAULT_CONFIG
flush.bytes                           262144      DEFAULT_CONFIG
flush.ms                              100         DEFAULT_CONFIG
initial.retention.local.target.bytes  -1          DEFAULT_CONFIG
initial.retention.local.target.ms     -1          DEFAULT_CONFIG
max.message.bytes                     1048576     DEFAULT_CONFIG
message.timestamp.type                CreateTime  DEFAULT_CONFIG
redpanda.iceberg.delete               true        DEFAULT_CONFIG
redpanda.iceberg.mode                 disabled    DEFAULT_CONFIG
redpanda.leaders.preference           none        DEFAULT_CONFIG
redpanda.remote.delete                true        DEFAULT_CONFIG
redpanda.remote.read                  true        DEFAULT_CONFIG
redpanda.remote.write                 true        DEFAULT_CONFIG
retention.bytes                       -1          DEFAULT_CONFIG
retention.local.target.bytes          -1          DEFAULT_CONFIG
retention.local.target.ms             86400000    DEFAULT_CONFIG
retention.ms                          604800000   DEFAULT_CONFIG
segment.bytes                         134217728   DEFAULT_CONFIG
segment.ms                            1209600000  DEFAULT_CONFIG
write.caching                         true        DEFAULT_CONFIG

Explore Data transforms

Data transforms let you run common data streaming tasks on the Redpanda broker, like filtering, scrubbing, and transcoding. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro.

This quickstart deployment comes with one transform function called regex running in your cluster. Its job is to find records in the logins topic that contain email addresses with the .edu domain and add those to a new topic called edu-filtered-domains.

In the left menu, click Transforms.

On the Transforms page, you see your transform. You can use Redpanda Console to manage and monitor your transforms.

The source code for the data transform function is in the docker-compose/transform/transform.go file that you downloaded. This file is commented to explain how it works.

View audit logs

Audit logs provide a record of all user actions. You can use these logs to track changes, troubleshoot issues, and maintain compliance with your organization’s security policies.

In the left menu, click Topics.
Click the Show internal topics checkbox.
Click the _redpanda.audit_log topic in the table.

Start streaming

To start building a basic streaming application, you can use the Redpanda CLI (rpk) to create a topic, produce messages to it, and consume messages from it. Each Redpanda broker comes preinstalled with rpk, so you can use it inside one of the Redpanda broker’s Docker containers.

For simplicity, this quickstart uses the bootstrapped superuser (superuser) to perform all operations. The superuser has full administrative privileges, making it convenient for setup and exploration. However, to enhance security in a production environment, Redpanda Data recommends creating users with restricted permissions for each task.

To use rpk inside the Redpanda broker’s Docker container:

Get information about the cluster:

docker exec -it redpanda-0 rpk cluster info -X user=superuser -X pass=secretpassword

bash

Create a topic called chat-room:

docker exec -it redpanda-0 rpk topic create chat-room \
  --replicas 3 \
  --topic-config redpanda.remote.read=true \
  --topic-config redpanda.remote.write=true \
  -X user=superuser \
  -X pass=secretpassword

bash

1	Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see Choose the replication factor.
2	Enable remote reads for this topic to read offloaded records from object storage.
3	Enable remote writes for this topic to offload older records to object storage. For more details, see Use Tiered Storage. Output: `TOPIC STATUS chat-room OK` bash

Produce a message to the topic:

docker exec -it redpanda-0 rpk topic produce chat-room -X user=superuser -X pass=secretpassword

bash

Enter a message, then press Enter:

Pandas are fabulous!

text

Example output:

Produced to partition 0 at offset 0 with timestamp 1663282629789.

Press Ctrl+C to finish producing messages to the topic.

Consume one message from the topic:

docker exec -it redpanda-0 rpk topic consume chat-room --num 1 -X user=superuser -X pass=secretpassword

bash

Your message is displayed along with its metadata:

{
  "topic": "chat-room",
  "value": "Pandas are fabulous!",
  "timestamp": 1663282629789,
  "partition": 0,
  "offset": 0
}

json

Connect to the cluster externally

It’s often more practical to connect to a remote cluster from your local machine with rpk. The docker-compose.yml file configured the containers to expose ports on your localhost, so you can communicate with the cluster outside the Docker network. To do so, download rpk on your local machine and configure it to connect to your cluster on the exposed ports.

Install rpk on your local machine:

Linux
macOS

amd64
arm64

curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip &&
  mkdir -p ~/.local/bin &&
  export PATH="~/.local/bin:$PATH" &&
  unzip rpk-linux-amd64.zip -d ~/.local/bin/

bash

curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip &&
  mkdir -p ~/.local/bin &&
  export PATH="~/.local/bin:$PATH" &&
  unzip rpk-linux-arm64.zip -d ~/.local/bin/

bash

You can use rpk on Windows only with WSL. However, commands that require Redpanda to be installed on your machine are not supported, such as rpk container commands, rpk iotune, and rpk redpanda commands.

If you don’t have Homebrew installed, install it.
To install or update rpk, run:
```
brew install redpanda-data/tap/redpanda
```
bash

Create a profile to configure rpk to connect to your cluster:
```
rpk profile create quickstart --from-profile rpk-profile.yaml
```
bash
The profile is configured in the docker-compose/rpk-profile.yaml file that you downloaded. This file is commented to explain how it works.

Get information about the cluster to test the connection:

rpk cluster info

bash

The Redpanda broker returns a list of all broker addresses, so rpk can communicate with all brokers directly. Each broker returns its configured advertise-* address that matches the port to which the initial connection has been made. You can see examples of these addresses in the Docker Compose file that you downloaded.

View offloaded data in Tiered Storage

Tiered Storage keeps local disk usage stable and performance consistent by offloading older data to object storage. You can also use Tiered Storage to recover clusters or topics, mount and unmount topics from a cluster, and more.

Redpanda supports Amazon S3, Azure ADLS, and Google GCS as object storage providers. For this quickstart, Redpanda uses MinIO as the object storage provider. MinIO is not supported for production deployments.

Check the size of remote and local disks for the chat-room topic you created earlier:

rpk cluster logdirs describe --topics chat-room

bash

BROKER  DIR                     TOPIC      PARTITION  SIZE  ERROR
0       /var/lib/redpanda/data  chat-room  0          611
0       remote://redpanda       chat-room  0          319
1       /var/lib/redpanda/data  chat-room  0          611
1       remote://redpanda       chat-room  0          319
2       /var/lib/redpanda/data  chat-room  0          611
2       remote://redpanda       chat-room  0          319

Local storage: /var/lib/redpanda/data holds recent data.
Remote storage: remote://redpanda holds offloaded data.

Open MinIO at http://localhost:9001/browser to view your data stored in the S3-compatible object store.

Login credentials:
- Username:
  minio
- Password:
  redpandaTieredStorage7

Deploy a pipeline with Redpanda Connect

Redpanda Connect is a powerful tool that allows you to build and manage data streaming pipelines. It supports hundreds of connectors for various systems, enabling seamless data ingestion, transformation, and routing between different sources and sinks. Using declarative YAML configuration files and the rpk CLI, you can quickly set up and deploy complex data pipelines in Redpanda.

Deploy the example pipeline:
```
rpk connect run ../generate-profiles.yaml
```
bash
This pipeline is configured to generate fake user profiles and produce them to the profiles topic in your Redpanda cluster. It’ll run until you stop it.

The pipeline configuration is in the docker-compose/generate-profiles.yaml file that you downloaded. This file is commented to explain how it works.
After a few seconds, stop the pipeline by pressing Ctrl+C.
Consume 10 messages from the profiles topic to confirm that the pipeline produced data to that topic.
```
rpk topic consume profiles --num 10
```
bash

Extend your 30-day trial

Your initial trial license is valid for 30 days. To extend it by another 30 days:

Sign up for a new trial license.
Upload the new license to your running cluster using rpk cluster license set:
```
rpk cluster license set <license>
```
bash
Replace the <license> placeholder with your license.

This command applies the new license to your Redpanda cluster, extending your access to enterprise features for another 30 days.
Restart Redpanda Console so that it can fetch the new license from the Redpanda brokers:
```
docker restart redpanda-console
```
bash

You can extend the trial license only once.

Upgrade to Redpanda Enterprise.

Clean up

To reset your environment, shut down the running processes and delete the containers:

docker compose down

bash

To delete the volumes along with all your cluster data:

docker compose down -v

bash

Customize the Docker quickstart

To customize this quickstart, you can configure Redpanda, Redpanda Console, or Redpanda Connect using the provided Docker Compose file. This allows you to tailor the services to fit your specific requirements.

Redpanda broker properties

To configure the Redpanda services with broker configuration properties, pass properties with the --set option in the redpanda start command in your Docker Compose file.

Example:

docker-compose.yml

redpanda-0: 
  command: 
    - redpanda
    - start
    - --set pandaproxy_client.retries=6

yaml

1	The service name in Docker Compose.
2	Overrides the default command for the Redpanda container.
3	Sets the `pandaproxy_client.retries` broker configuration property to `6`.

Redpanda cluster properties

Cluster-level configurations are defined in the bootstrap.yml file. This file is included in your Docker Compose setup. It contains essential settings that apply to the entire Redpanda cluster. You can find all cluster properties in Cluster Configuration Properties.

Example:

docker-compose.yml

redpanda:
  volumes:
    - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml

yaml

1	Mounts the `bootstrap.yml` file from your local parent directory to the container’s `/etc/redpanda/` directory.

Redpanda Console configuration

Redpanda Console configuration is managed with the environment variable CONSOLE_CONFIG_FILE. You can specify the path to your configuration file in the Docker Compose file.

Example:

docker-compose.yml

console:
  environment:
    CONSOLE_CONFIG_FILE: |
      # Configure a connection to the Redpanda cluster
      # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/
      kafka:
        brokers: ["redpanda-0:9092"]
        schemaRegistry:
          enabled: true
          urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"]
        sasl:
          enabled: true
          username: superuser
          password: secretpassword
          mechanism: SCRAM-SHA-256

yaml

Redpanda Connect configuration

Existing data pipelines are configured using the CONNECT_CFG_FILE environment variable in the connect service. This environment variable contains YAML configuration settings for Redpanda Connect.

Example:

docker-compose.yml

connect:
  environment:
    CONNECT_CFG_FILE: |
      input:
        generate:
          interval: 1s
          mapping: |
            let first_name = fake("first_name")
            let last_name  = fake("last_name")

            root.user_id    = counter()
            root.name       = $$first_name + " " + $$last_name
            root.email      = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase()
            root.ip         = fake("ipv4")
            root.login_time = now()
      pipeline:
        processors:
          - mapping: |
              root = range(0, random_int(min:2, max:4)).map_each(cust -> this)
          - unarchive:
              format: "json_array"
          - mapping: |
              if batch_index() == 0 {
                meta topic = "logins"
                root = this
              } else {
                meta topic       = "transactions"
                root.user_id     = this.user_id
                root.email       = this.email
                root.index       = batch_index() - 1
                root.product_url = fake("url")
                root.price       = fake("amount_with_currency")
                root.timestamp   = now()
              }
      output:
        kafka_franz:
          seed_brokers: [ "redpanda-0:9092" ]
          topic: $${! metadata("topic") }
          sasl:
            - mechanism: SCRAM-SHA-256
              password: secretpassword
              username: superuser

yaml

To create new data pipelines, run rpk connect run, specifying the path to your own YAML configuration file:

rpk connect run <path-to-config>

bash

Next steps

Contact Redpanda to discuss using Redpanda Enterprise Edition in production.

What do you like about this doc?

Let us know what we do well:

Let us contact you about your feedback:

What did you not like about this doc?

Let us know what we can improve:

Let us contact you about your feedback:

Redpanda Self-Managed Quickstart

Enterprise features

Prerequisites

Deploy Redpanda Self-Managed

Explore Redpanda Console

Log in to Redpanda Console

View topics and filter messages

Explore Schema Registry

Configure access control for Redpanda

Explore Data transforms

View audit logs

Start streaming

Connect to the cluster externally

View offloaded data in Tiered Storage

Deploy a pipeline with Redpanda Connect

Extend your 30-day trial

Clean up

Customize the Docker quickstart

Redpanda broker properties

Redpanda cluster properties

Redpanda Console configuration

Redpanda Connect configuration

Next steps

Suggested reading

Simple online edits

Contribution guide