Skip to main content
Version: 21.11

Kubernetes Quick Start Guide

With Redpanda you can get up and running with streaming quickly and be fully compatible with the Kafka ecosystem.

This quick start guide can help you get started with Redpanda for development and testing purposes. To get up and running you need to create a cluster and deploy the Redpanda operator on the cluster.

Note - Run a container inside the Kubernetes cluster to communicate with the Redpanda cluster. Currently, a load balancer is not automatically created during deployment by default.

Note - In the steps below, the .yaml file that you use to install Redpanda sets developerMode: true. If you want to set developerMode: false, for optimal configuration it is recommended that you run rpk redpanda tune all directly on the host before you create a Redpanda cluster. You can find more information about the command as well as tuning recommendations in the Set Redpanda production mode documentation. If rpk is not available, verify that fs.aio-max-nr is set to 1048576 or greater. You can set fs.aio-max-nr by running sysctl -w fs.aio-max-nr=1048576.

Prerequisites

Before you start installing Redpanda you need to setup your Kubernetes environment.

Install Kubernetes, Helm, and cert-manager

You'll need to install:

  • Kubernetes v1.19 or above

  • kubectl v1.19 or above

  • helm v3.0.0 or above

  • cert-manager v1.5.0 or above

    Follow the instructions to verify that cert-manager is ready to create certificates.

Make sure you also have these common tools installed:

To run locally

note

Make sure that you have kind configured in your path. This reference in the GO documentation can help you configure the path.

Create a Kubernetes cluster

You can either create a Kubernetes cluster on your local machine or on a cloud provider.

Kind is a tool that lets you create local Kubernetes clusters using Docker. After you install Kind, verify that port 30001 is free on your machine before you set up a cluster or cluster creation will fail. Set up a cluster with:

kind create cluster --config docs/kind-external.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30001
hostPort: 30001

Kubectl context

Most cloud utility tools will automatically change your kubectl config file.
To check if you're in the correct context, run the command:

kubectl config current-context

For Digital Ocean for example, the output will look similar to this:

do-nyc1-redpanda

If you're running multiple clusters or if the config file wasn't set up automatically, look for more information in the Kubernetes documentation.

Install cert-manager

The Redpanda operator requires cert-manager to create certificates for TLS communication. You can install cert-manager with a CRD, but here's the command to install using helm:

helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.8.0 \
--set installCRDs=true

We recommend that you use the verification procedure in the cert-manager docs to verify that cert-manager is working correcly.

Use Helm to install Redpanda operator

  1. Using Helm, add the Redpanda chart repository and update it:

    helm repo add redpanda https://charts.vectorized.io/ && \
    helm repo update
  2. Just to simplify the commands, create a variable to hold the latest version number:

    export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)
    note

    You can find information about the versions of the operator in the list of operator releases.
    We're using jq to help us. If you don't have it installed run this command:

    sudo apt-get update && \
    sudo apt-get install jq
  3. Install the Redpanda operator CRD:

    kubectl apply \
    -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION
  4. Install the Redpanda operator on your Kubernetes cluster with:

    helm install \
    redpanda-operator \
    redpanda/redpanda-operator \
    --namespace redpanda-system \
    --create-namespace \
    --version $VERSION

Install and connect to a Redpanda cluster

After you set up Redpanda in your Kubernetes cluster, you can use our samples to install a cluster and see Redpanda in action.

Let's try setting up a Redpanda topic to handle a stream of events from a chat application with 5 chat rooms:

  1. Create a namespace for your cluster:

    kubectl create ns chat-with-me
  2. Install a cluster from our sample files, for example the single-node cluster:

    kubectl apply \
    -n chat-with-me \
    -f https://raw.githubusercontent.com/redpanda-data/redpanda/dev/src/go/k8s/config/samples/one_node_external.yaml

    You can see the resource configuration options in the cluster_types file.

Check etc/hosts file

Verify that 0.local.rp is mapped to 127.0.0.1 on your system. It will contain a line similar to this:

127.0.0.1 0.local.rp

Do some streaming

Here are some sample commands to produce and consume streams:

  1. Create a topic. We'll call it "twitch_chat":

    rpk topic create twitch_chat --brokers=0.local.rp:30001
  2. Produce messages to the topic:

    rpk topic produce twitch_chat --brokers=0.local.rp:30001

    Type text into the topic and press Ctrl + D to separate between messages.

    Press Ctrl + C to exit the produce command.

  3. Consume (or read) the messages in the topic:

    rpk topic consume twitch_chat --brokers=0.local.rp:30001

    Each message is shown with its metadata, like this:

    {
    "message": "How do you stream with Redpanda?\n",
    "partition": 0,
    "offset": 1,
    "timestamp": "2021-02-10T15:52:35.251+02:00"
    }

Next steps

  • Check out our in-depth explanation of how to connect external clients to a Redpanda Kubernetes deployment.
  • Contact us in our Slack community so we can work together to implement your Kubernetes use cases.