Docker

Docker Quick Start Guide

Redpanda is a modern streaming platform for mission critical workloads. With Redpanda you can get up and running with streaming quickly and be fully compatible with the Kafka ecosystem.

This quick start guide can help you get started with Redpanda for development and testing purposes. For production or benchmarking, setup a production deployment.

caution

Running Redpanda directly via Docker is not supported for production usage. This platform should only be used for testing.

Get your cluster ready

To get a cluster ready for streaming, either run a single docker container with Redpanda running or a cluster of 3 containers.

note

• You can also use rpk container to run Redpanda in containers without having to interact with Docker at all.

Single command for a 1-node cluster

With a 1-node cluster you can test out a simple implementation of Redpanda.

Notes

• --overprovisioned is used to accommodate docker resource limitations.
• --pull=always makes sure that you are always working with the latest version.

docker run -d --pull=always --name=redpanda-1 --rm \
-p 8081:8081 \
-p 8082:8082 \
-p 9092:9092 \
-p 9644:9644 \
docker.redpanda.com/redpandadata/redpanda:latest \
redpanda start \
--overprovisioned \
--smp 1  \
--memory 1G \
--reserve-memory 0M \
--node-id 0 \
--check=false

You can do some simple topic actions to do some streaming. Otherwise, just point your Kafka-compatible client to 127.0.0.1:9092.

Set up a 3-node cluster

To test out the interaction between nodes in a cluster, set up a Docker network with 3 containers in a cluster.

Create network and persistent volumes

First we need to set up a bridge network so that the Redpanda instances can communicate with each other but still allow for the Kafka API to be available on the localhost. We'll also create the persistent volumes that let the Redpanda instances keep state during instance restarts.

docker network create -d bridge redpandanet && \
docker volume create redpanda1 && \
docker volume create redpanda2 && \
docker volume create redpanda3

Start Redpanda nodes

We then need to start the nodes for the Redpanda cluster.

docker run -d \
--pull=always \
--name=redpanda-1 \
--hostname=redpanda-1 \
--net=redpandanet \
-p 8081:8081 \
-p 8082:8082 \
-p 9092:9092 \
-p 9644:9644 \
-v "redpanda1:/var/lib/redpanda/data" \
docker.redpanda.com/redpandadata/redpanda redpanda start \
--smp 1  \
--memory 1G  \
--reserve-memory 0M \
--overprovisioned \
--node-id 0 \
--check=false \
--pandaproxy-addr INSIDE://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082 \
--advertise-pandaproxy-addr INSIDE://redpanda-1:28082,OUTSIDE://localhost:8082 \
--kafka-addr INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 \
--advertise-kafka-addr INSIDE://redpanda-1:29092,OUTSIDE://localhost:9092 \
--rpc-addr 0.0.0.0:33145 \
--advertise-rpc-addr redpanda-1:33145 && \

docker run -d \
--pull=always \
--name=redpanda-2 \
--hostname=redpanda-2 \
--net=redpandanet \
-p 8083:8083 \
-p 9093:9093 \
-v "redpanda2:/var/lib/redpanda/data" \
docker.redpanda.com/redpandadata/redpanda redpanda start \
--smp 1  \
--memory 1G  \
--reserve-memory 0M \
--overprovisioned \
--node-id 1 \
--seeds "redpanda-1:33145" \
--check=false \
--pandaproxy-addr INSIDE://0.0.0.0:28083,OUTSIDE://0.0.0.0:8083 \
--advertise-pandaproxy-addr INSIDE://redpanda-2:28083,OUTSIDE://localhost:8083 \
--kafka-addr INSIDE://0.0.0.0:29093,OUTSIDE://0.0.0.0:9093 \
--advertise-kafka-addr INSIDE://redpanda-2:29093,OUTSIDE://localhost:9093 \
--rpc-addr 0.0.0.0:33146 \
--advertise-rpc-addr redpanda-2:33146 && \

docker run -d \
--pull=always \
--name=redpanda-3 \
--hostname=redpanda-3 \
--net=redpandanet \
-p 8084:8084 \
-p 9094:9094 \
-v "redpanda3:/var/lib/redpanda/data" \
docker.redpanda.com/redpandadata/redpanda redpanda start \
--smp 1  \
--memory 1G  \
--reserve-memory 0M \
--overprovisioned \
--node-id 2 \
--seeds "redpanda-1:33145" \
--check=false \
--pandaproxy-addr INSIDE://0.0.0.0:28084,OUTSIDE://0.0.0.0:8084 \
--advertise-pandaproxy-addr INSIDE://redpanda-3:28084,OUTSIDE://localhost:8084 \
--kafka-addr INSIDE://0.0.0.0:29094,OUTSIDE://0.0.0.0:9094 \
--advertise-kafka-addr INSIDE://redpanda-3:29094,OUTSIDE://localhost:9094 \
--rpc-addr 0.0.0.0:33147 \
--advertise-rpc-addr redpanda-3:33147

Now you can run rpk on one of the containers to interact with the cluster:

docker exec -it redpanda-1 rpk cluster info

Or as a separate container in the same network:

docker run --net redpandanet docker.redpanda.com/redpandadata/redpanda cluster info --brokers=redpanda-1:29092

The output of the status command looks like:

BROKERS
=======
ID    HOST        PORT
0*    redpanda-1  29092
1     redpanda-2  29093
2     redpanda-3  29094

Bring up a docker-compose file

You can easily try out different docker configuration parameters with a docker-compose file.

1. Save this content as docker-compose.yml:

version: '3.7'
services:
  redpanda:
    # NOTE: Please use the latest version here!
    image: docker.redpanda.com/redpandadata/redpanda:v21.11.15
    container_name: redpanda-1
    command:
    - redpanda
    - start
    - --smp
    - '1'
    - --reserve-memory
    - 0M
    - --overprovisioned
    - --node-id
    - '0'
    - --kafka-addr
    - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
    - --advertise-kafka-addr
    - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    - --pandaproxy-addr
    - PLAINTEXT://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082
    - --advertise-pandaproxy-addr
    - PLAINTEXT://redpanda:28082,OUTSIDE://localhost:8082
    ports:
    - 8081:8081
    - 8082:8082
    - 9092:9092
    - 28082:28082
    - 29092:29092

2. In the directory where the file is saved, run:

   docker-compose up -d

If you want to change the parameters, edit the docker-compose file and run the command again.

Do some streaming

Here are some sample commands to produce and consume streams:

1. Create a topic. We'll call it "twitch_chat":

   docker exec -it redpanda-1 \
   rpk topic create twitch_chat --brokers=localhost:9092

2. Produce messages to the topic:

   docker exec -it redpanda-1 \
   rpk topic produce twitch_chat --brokers=localhost:9092

   Type text into the topic and press Ctrl + D to separate between messages.

   Press Ctrl + C to exit the produce command.

3. Consume (or read) the messages in the topic:

   docker exec -it redpanda-1 \
   rpk topic consume twitch_chat --brokers=localhost:9092

   Each message is shown with its metadata, like this:

   {
   "message": "How do you stream with Redpanda?\n",
   "partition": 0,
   "offset": 1,
   "timestamp": "2021-02-10T15:52:35.251+02:00"
   }

You've just installed Redpanda and done streaming in a few easy steps.

Clean Up

When you are finished with the cluster, you can shutdown and delete the containers. Change the commands below accordingly if you used the 1-cluster option, or the 3-cluster option.

docker stop redpanda-1 redpanda-2 redpanda-3 && \
docker rm redpanda-1 redpanda-2 redpanda-3

If you set up volumes and a network, delete them with:

docker volume rm redpanda1 redpanda2 redpanda3 && \
docker network rm redpandanet

What's Next?

• Our FAQ page shows all of the clients that you can use to do streaming with Redpanda. (Spoiler: Any Kafka-compatible client!)
• Get a multi-node cluster up and running using rpk container.
• Want to setup a production cluster? Check out our Production Deployment Guide.