Redpanda Quickstart

Spin up a Redpanda cluster, create a basic streaming application, and explore your cluster in Redpanda Console.

Choose one of the following options:

• Redpanda Cloud - Dedicated: Use a fully managed Redpanda cluster hosted on dedicated resources and configurable instance types in Redpanda Cloud.
• Docker (self-hosted): Evaluate Redpanda on your Linux, macOS, or Windows machine.
  important

  Redpanda in Docker is supported only for development and testing. To deploy self-hosted Redpanda in production, use one of the following environments:

  • Linux
  • Kubernetes

Prerequisites

Redpanda Cloud

Sign up for a trial of Redpanda Cloud - Dedicated Clusters.

Docker

You need Docker Compose. For installation instructions, see the Docker Compose documentation.

To check if you have Docker Compose installed, run the following:

docker compose version

You should see the version of Docker Compose that's installed on your local machine.

Deploy Redpanda

Redpanda Cloud

To deploy a Redpanda cluster in the Dedicated cloud, create a namespace, then follow the steps for creating a Dedicated cluster.

To create a namespace:

1. Go to Redpanda Cloud and click Log in.

2. Enter your organization name and click Continue.

3. Click Add namespace.

   Namespaces help you organize clusters by owner, team, or project for example.

4. Enter a name for your namespace.

To create a cluster:

1. Click Add Cluster.

2. Click Create Dedicated cluster, and enter the following values:

   • Cluster name:

     redpandaquickstart

   • Throughput: 1 MBps

   • Version: Choose the latest available version

3. Click Next to see the Cloud settings options, and enter the following values:

   • Provider: AWS

   • Region: Any region that's closest to you

   • Availability: Single AZ

   • Select a zone: Choose any zone

4. Click Next to see the Networking options, and enter the following values:

   • Connection Type: Public Network

5. Click Create.

6. Wait for both your network and your cluster to be created.

Docker

For lightweight testing, you can start a single Redpanda broker, or you can use a more robust example with three Redpanda brokers. With three brokers, you can configure your topics with a replication factor of three so that the cluster can recover from a single-broker failure.

Single Broker

1. Copy and paste the following YAML content into a file named docker-compose.yml on your local filesystem.

   Reveal the YAML content

   docker-compose.yml

   ---
   version: "3.7"
   name: redpanda-quickstart
   networks:
     redpanda_network:
       driver: bridge
   volumes:
     redpanda-0: null
   services:
     redpanda-0:
       command:
         - redpanda
         - start
         - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
         # Address the broker advertises to clients that connect to the Kafka API.
         # Use the internal addresses to connect to the Redpanda brokers'
         # from inside the same Docker network.
         # Use the external addresses to connect to the Redpanda brokers'
         # from outside the Docker network.
         - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
         - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
         # Address the broker advertises to clients that connect to the HTTP Proxy.
         - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
         - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
         # Redpanda brokers use the RPC API to communicate with eachother internally.
         - --rpc-addr redpanda-0:33145
         - --advertise-rpc-addr redpanda-0:33145
         # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
         - --smp 1
         # The amount of memory to make available to Redpanda.
         - --memory 1G
         # Mode dev-container uses well-known configuration properties for development in containers.
         - --mode dev-container
         # enable logs for debugging.
         - --default-log-level=debug
       image: docker.redpanda.com/redpandadata/redpanda:v23.1.2
       container_name: redpanda-0
       volumes:
         - redpanda-0:/var/lib/redpanda/data
       networks:
         - redpanda_network
       ports:
         - 18081:18081
         - 18082:18082
         - 19092:19092
         - 19644:9644
     console:
       container_name: redpanda-console
       image: docker.redpanda.com/vectorized/console:v2.2.2
       networks:
         - redpanda_network
       entrypoint: /bin/sh
       command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
       environment: 
         CONFIG_FILEPATH: /tmp/config.yml
         CONSOLE_CONFIG_FILE: |
           kafka:
             brokers: ["redpanda-0:9092"]
             schemaRegistry:
               enabled: true
               urls: ["http://redpanda-0:8081"]
           redpanda:
             adminApi:
               enabled: true
               urls: ["http://redpanda-0:9644"]
       ports:
         - 8080:8080
       depends_on:
         - redpanda-0

2. Run the following in the directory where you saved the docker-compose.yml file:

   docker compose up -d
   Example output

   [+] Running 4/4
   ⠿ Network redpanda_network                 Created  0.0s
   ⠿ Volume "redpanda-quickstart_redpanda-0"  Created  0.0s
   ⠿ Container redpanda-0                     Started  0.3s
   ⠿ Container redpanda-console               Started  0.6s

Three Brokers

1. Copy and paste the following YAML content into a file named docker-compose.yml on your local filesystem.

   Reveal the YAML content

   docker-compose.yml

   ---
   version: "3.7"
   name: redpanda-quickstart
   networks:
     redpanda_network:
       driver: bridge
   volumes:
     redpanda-0: null
     redpanda-1: null
     redpanda-2: null
   services:
     redpanda-0:
       command:
         - redpanda
         - start
         - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
         # Address the broker advertises to clients that connect to the Kafka API.
         # Use the internal addresses to connect to the Redpanda brokers'
         # from inside the same Docker network.
         # Use the external addresses to connect to the Redpanda brokers'
         # from outside the Docker network.
         - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
         - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
         # Address the broker advertises to clients that connect to the HTTP Proxy.
         - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
         - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
         # Redpanda brokers use the RPC API to communicate with eachother internally.
         - --rpc-addr redpanda-0:33145
         - --advertise-rpc-addr redpanda-0:33145
         # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
         - --smp 1
         # The amount of memory to make available to Redpanda.
         - --memory 1G
         # Mode dev-container uses well-known configuration properties for development in containers.
         - --mode dev-container
         # enable logs for debugging.
         - --default-log-level=debug
       image: docker.redpanda.com/redpandadata/redpanda:v23.1.2
       container_name: redpanda-0
       volumes:
         - redpanda-0:/var/lib/redpanda/data
       networks:
         - redpanda_network
       ports:
         - 18081:18081
         - 18082:18082
         - 19092:19092
         - 19644:9644
     redpanda-1:
       command:
         - redpanda
         - start
         - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092
         - --advertise-kafka-addr internal://redpanda-1:9092,external://localhost:29092
         - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:28082
         - --advertise-pandaproxy-addr internal://redpanda-1:8082,external://localhost:28082
         - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:28081
         - --rpc-addr redpanda-1:33145
         - --advertise-rpc-addr redpanda-1:33145
         - --smp 1
         - --memory 1G
         - --mode dev-container
         - --default-log-level=debug
         - --seeds redpanda-0:33145
       image: docker.redpanda.com/redpandadata/redpanda:v23.1.2
       container_name: redpanda-1
       volumes:
         - redpanda-1:/var/lib/redpanda/data
       networks:
         - redpanda_network
       ports:
         - 28081:28081
         - 28082:28082
         - 29092:29092
         - 29644:9644
       depends_on:
         - redpanda-0
     redpanda-2:
       command:
         - redpanda
         - start
         - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:39092
         - --advertise-kafka-addr internal://redpanda-2:9092,external://localhost:39092
         - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:38082
         - --advertise-pandaproxy-addr internal://redpanda-2:8082,external://localhost:38082
         - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:38081
         - --rpc-addr redpanda-2:33145
         - --advertise-rpc-addr redpanda-2:33145
         - --smp 1
         - --memory 1G
         - --mode dev-container
         - --default-log-level=debug
         - --seeds redpanda-0:33145
       image: docker.redpanda.com/redpandadata/redpanda:v23.1.2
       container_name: redpanda-2
       volumes:
         - redpanda-2:/var/lib/redpanda/data
       networks:
         - redpanda_network
       ports:
         - 38081:38081
         - 38082:38082
         - 39092:39092
         - 39644:9644
       depends_on:
         - redpanda-0
     console:
       container_name: redpanda-console
       image: docker.redpanda.com/redpandadata/console:v2.2.2
       networks:
         - redpanda_network
       entrypoint: /bin/sh
       command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
       environment: 
         CONFIG_FILEPATH: /tmp/config.yml
         CONSOLE_CONFIG_FILE: |
           kafka:
             brokers: ["redpanda-0:9092"]
             schemaRegistry:
               enabled: true
               urls: ["http://redpanda-0:8081"]
           redpanda:
             adminApi:
               enabled: true
               urls: ["http://redpanda-0:9644"]
       ports:
         - 8080:8080
       depends_on:
         - redpanda-0

2. Run the following in the directory where you saved the docker-compose.yml file:

   docker compose up -d
   Example output

   [+] Running 8/8
   ⠿ Network redpanda_network                  Created  0.0s
   ⠿ Volume "redpanda-quickstart_redpanda-0"   Created  0.0s
   ⠿ Volume "redpanda-quickstart_redpanda-1"   Created  0.0s
   ⠿ Volume "redpanda-quickstart_redpanda-2"   Created  0.0s
   ⠿ Container redpanda-0                      Started  0.3s
   ⠿ Container redpanda-console                Started  0.6s
   ⠿ Container redpanda-1                      Started  0.7s
   ⠿ Container redpanda-2                      Started  0.7s

Start streaming

To start building a basic streaming application, you can use rpk to create a topic, produce messages to it, and consume messages from it. rpk is a CLI tool for connecting to and interacting with Redpanda brokers.

Redpanda Cloud

1. Install rpk on your local machine:

   note

   The rpk binary is not supported on Windows.

   Linux

   1. Download the rpk archive for Linux:

      curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip

   2. Ensure that you have the folder ~/.local/bin:

      mkdir -p ~/.local/bin

   3. Add it to your $PATH:

      export PATH="~/.local/bin:$PATH"

   4. Unzip the rpk files to your ~/.local/bin/ directory:

      unzip rpk-linux-amd64.zip -d ~/.local/bin/

   5. Run rpk version to display the rpk binary version:

      rpk version
      Example output

      v23.1.2 (rev 6002d4d)

   macOS

   1. If you don't have Homebrew installed, install it.

   2. Install rpk:

      brew install redpanda-data/tap/redpanda

   3. Run rpk version to display the rpk binary version:

      rpk version
      Example output

      v23.1.2 (rev 6002d4d)

2. On the Overview page of Redpanda Cloud, copy your bootstrap server address and set it as an environment variable on your local machine:

   export REDPANDA_BROKERS="<bootstrap-server-address>"

3. In Redpanda Cloud, go to Security and create a user called redpanda-twitch-account.

4. Copy the password, and mechanism, and set the following environment variables on your local machine:

   export REDPANDA_SASL_USERNAME="redpanda-twitch-account"
   export REDPANDA_SASL_PASSWORD="<password>"
   export REDPANDA_SASL_MECHANISM="<mechanism>"

5. In Redpanda Cloud, click the name of your user and add the following permissions to the ACL (access control list):

   • Host: *

   • Topic name:

     twitch_chat

   • Operations: All

6. Use rpk on your local machine to authenticate to Redpanda as the redpanda-twitch-account user and get information about the cluster:

   rpk cluster info --tls-enabled
   Example output

   CLUSTER
   =======
   redpanda.rp-example-cluster-id
   
   BROKERS
   =======
   ID    HOST                                  PORT   RACK
   0*    0-example.fmc.prd.cloud.redpanda.com  30092  euw1-az1
   1     1-example.fmc.prd.cloud.redpanda.com  30092  euw1-az1
   2     2-example.fmc.prd.cloud.redpanda.com  30092  euw1-az1

7. Create a topic called twitch_chat.

   You granted permissions to the redpanda-twitch-account user to access only this topic.

   rpk topic create twitch_chat --tls-enabled
   Example output

   TOPIC       STATUS
   twitch_chat  OK

8. Produce a message to the topic:

   rpk topic produce twitch_chat --tls-enabled

9. Enter a message, then press Enter:

   Pandas are fabulous!
   Example output

   Produced to partition 0 at offset 0 with timestamp 1663282629789.

10. Press Ctrl+C to finish producing messages to the topic.

11. Consume one message from the topic:

    rpk topic consume twitch_chat --num 1 --tls-enabled
    Example output

    Your message is displayed along with its metadata,:

    {
      "topic": "twitch_chat",
      "value": "Pandas are fabulous!",
      "timestamp": 1663282629789,
      "partition": 0,
      "offset": 0
    }

Docker

Each Redpanda broker comes preinstalled with rpk, so you can use it inside one of the Redpanda broker's Docker containers or you can use install it on your local machine to test external connectivity. The docker-compose.yml file configured the containers to expose ports on your localhost so that you can communicate with the cluster outside the Docker network.

To use rpk inside the Redpanda broker's Docker container:

1. Get information about the cluster.

   docker exec -it redpanda-0 rpk cluster info
   Example output

   BROKERS
   =======
   ID    HOST        PORT
   0*    172.24.1.2  9092
   1     172.24.1.3  9092
   2     172.24.1.4  9092

2. Create a topic called twitch_chat:

   docker exec -it redpanda-0 rpk topic create twitch_chat
   Example output

   TOPIC       STATUS
   twitch_chat  OK

3. Produce a message to the topic:

   docker exec -it redpanda-0 rpk topic produce twitch_chat

4. Enter a message, then press Enter:

   Pandas are fabulous!
   Example output

   Produced to partition 0 at offset 0 with timestamp 1663282629789.

5. Press Ctrl+C to finish producing messages to the topic.

6. Consume one message from the topic:

   docker exec -it redpanda-0 rpk topic consume twitch_chat --num 1
   Example output

   Your message is displayed along with its metadata,:

   {
     "topic": "twitch_chat",
     "value": "Pandas are fabulous!",
     "timestamp": 1663282629789,
     "partition": 0,
     "offset": 0
   }

To test external connectivity using your local machine:

1. Install rpk on your local machine:

   note

   The rpk binary is not supported on Windows.

   Linux

   1. Download the rpk archive for Linux:

      curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip

   2. Ensure that you have the folder ~/.local/bin:

      mkdir -p ~/.local/bin

   3. Add it to your $PATH:

      export PATH="~/.local/bin:$PATH"

   4. Unzip the rpk files to your ~/.local/bin/ directory:

      unzip rpk-linux-amd64.zip -d ~/.local/bin/

   5. Run rpk version to display the rpk binary version:

      rpk version
      Example output

      v23.1.2 (rev 6002d4d)

   macOS

   1. If you don't have Homebrew installed, install it.

   2. Install rpk:

      brew install redpanda-data/tap/redpanda

   3. Run rpk version to display the rpk binary version:

      rpk version
      Example output

      v23.1.2 (rev 6002d4d)

2. Get information about the cluster to test the connection.

   rpk cluster info --brokers 127.0.0.1:19092
   Example output

   BROKERS
   =======
   ID    HOST        PORT
   0*    127.0.0.1   19092
   1     127.0.0.1   29092
   2     127.0.0.1   39092
   
   TOPICS
   ======
   NAME         PARTITIONS  REPLICAS
   twitch_chat  1           1
   note

   The Redpanda broker returns a list of all broker addresses so that rpk can communicate with all brokers directly. Each broker returns its configured advertise-* address that matches the port to which the initial connection has been made.

Explore your topic

Redpanda Cloud

In Redpanda Cloud, go to Topics > twitch_chat.

The message that you produced to the topic is displayed along with some other details about the topic.

Docker

To explore your topic, you can use Redpanda Console, which you deployed in the docker-compose.yml file and is running locally on port 8080. Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications.

1. Open Redpanda Console in a web browser.

   All your Redpanda brokers are listed along with their IP addresses and IDs.

2. Go to Topics > twitch_chat.

   The message that you produced to the topic is displayed along with some other details about the topic.

Clean up

Redpanda Cloud

When you're finished with the cluster, you can delete it.

Go to Cluster settings and click Delete cluster.

Docker

When you're finished with the cluster, shut down and delete the containers:

docker compose down

To delete the volumes along with all your cluster data, run the following:

docker compose down -v

Next steps

Redpanda Cloud

• Build a sample application.
• Learn more about rpk.
• Learn more about Redpanda Cloud.

Docker

• Build a sample application.
• Learn more about Redpanda Console.
• Learn more about rpk.
• Deploy to production.
• Try more Docker Compose examples.
• Find the Docker images for Redpanda on Docker Hub.
• Find the Docker images for Redpanda Console on Docker Hub.

Suggested reading

Explore the rpk commands that you used in this quickstart:

• rpk cluster
• rpk topic create
• rpk topic produce
• rpk topic consume