Install Redpanda for Kubernetes with kind

Deprecated red

The Cluster and Console resources are deprecated. For details, see the deprecation notice. To migrate to the Redpanda resource, see Migrate from Cluster and Console Custom Resources

This guide helps you set up Redpanda locally for development and testing purposes on macOS, Windows, or Linux. Start here to set up Redpanda quickly to try it out or for CI/CD purposes.

This guide uses kind to create a local Kubernetes cluster in which Redpanda is deployed. It also verifies that the Kafka API port is exposed to your local network so that you can interact with Redpanda from your local machine and not just from inside the Kubernetes cluster.

To get up and running, you create a cluster and deploy the Redpanda operator on the cluster. The steps limit the number of nodes to one, because multi-node local clusters are not currently supported. For details on how to access Redpanda outside the Kubernetes network, see Connecting Remotely to Kubernetes.

In these steps, the YAML file you use to install Redpanda sets developerMode: true. If you choose to set developerMode: false, run rpk redpanda tune all directly on the host before you create a Redpanda cluster. This command sets tuning parameters for optimal configuration. For more information, see Set Redpanda production mode. If rpk is not available, verify that fs.aio-max-nr is set to 1048576 or greater. You can set fs.aio-max-nr by running sysctl -w fs.aio-max-nr=1048576.

Prerequisites

Before you install Redpanda, verify that you have the following software installed on the machine where you want to run Redpanda:

  • kubectl - version 1.21 or later

  • Helm - version 3.0.0 or later

  • kind - version 0.11.1 or later

  • Go - v1.17 or later (only required for Windows setup)

  • jq - This setup uses jq to set the Redpanda $VERSION environment variable. (If you don’t have jq installed, you can run the same commands by replacing the $VERSION environment variable with the version of Redpanda that you’re using.) Run the one of the following commands to install jq:

  • brew

  • apt

brew install jq
sudo apt-get update && \
sudo apt-get install jq

Create a Kubernetes cluster

Use kind to create a Kubernetes cluster on your local machine. kind lets you create local Kubernetes clusters using Docker. After you install kind, verify that port 30001 is free on your machine before you set up a cluster, or cluster creation will fail.

To enable access from your operating system to the cluster, create a kind-external.yaml file that contains the following text:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
    - containerPort: 30001
      hostPort: 30001

Set up a cluster:

kind create cluster --config kind-external.yaml

Install cert-manager

The Redpanda operator requires cert-manager to create certificates for TLS communication. You can install cert-manager with Helm or kubectl.

Before you install cert-manager, verify that the version you’re installing is compatible with your Kubernetes version. The cert-manager Supported Releases documentation lists which Kubernetes versions are supported.

To install cert-manager with Helm:

helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.12.2 \
  --set installCRDs=true

Verify cert-manager installation

Use the verification procedure in the cert-manager documentation to verify that cert-manager is deployed correctly.

Install rpk

Redpanda Keeper, or rpk, is a CLI utility that you use to interact with Redpanda.

Install rpk on the host machine:

brew install redpanda-data/tap/redpanda
On Windows, you must compile rpk yourself with Go. You can find the rpk source files in GitHub under redpanda/src/go/rpk.

Install Redpanda operator with Helm

  1. Use Helm to add the Redpanda chart repository and update it:

    helm repo add vectorized https://charts.vectorized.io/ && \
helm repo update

  2. Set the $VERSION environment variable to the latest operator and Redpanda version:

    export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)

    If you prefer to manually enter the Redpanda version, or if you want to verify the version, the following command returns the installed version:

    curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | grep tag_name

    Find information about the versions of the operator in the list of operator releases.

  3. Install the Redpanda operator CRD with the bash or zsh:

    • bash

    • zsh

    kubectl apply \
-k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION
    noglob kubectl apply \
-k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION

  4. Install the Redpanda operator on your Kubernetes cluster:

    helm install \
redpanda-operator \
vectorized/redpanda-operator \
--namespace redpanda-system \
--create-namespace \
--version $VERSION

Install and connect to a Redpanda cluster

After you set up Redpanda in your Kubernetes cluster, you can use the sample configuration files in GitHub to install a cluster and see Redpanda in action.

This example is an imaginary chat application, panda-chat, but you can replace panda-chat with any string. In this example, panda-chat has five chat rooms.

Complete the following steps to manage a stream of events from panda-chat:

  1. Create a namespace for the cluster:

    kubectl create ns panda-chat

  2. Install a single-node cluster like this (note that only single-node clusters are supported for local access clusters):

    kubectl apply \
-n panda-chat \
-f https://raw.githubusercontent.com/redpanda-data/redpanda/dev/src/go/k8s/config/samples/one_node_external.yaml

You can view the resource configuration options, such as storage capacity, network configuration, or TLS configuration in the cluster_types file in GitHub. You can also find additional sample configuration files.

Check the /etc/hosts file

Map 0.local.rp to 127.0.0.1 on the host machine that runs the kind cluster. The /etc/hosts file contains a line similar to this:

127.0.0.1 0.local.rp

The local.rp address is specified in the one_node_external.yaml file that you used in the previous step.

If you’re running Windows, this is the location of the file: C:\Windows\System32\drivers\etc\hosts

Start streaming

Use rpk to run commands. rpk is a CLI tool you can use to work with your Redpanda nodes. See rpk Commands. Here are some sample commands to produce and consume streams:

Create a panda-chat topic with five partitions:

rpk topic create panda-chat -p 5 --brokers localhost:30001

Produce messages to the topic:

rpk topic produce panda-chat --brokers localhost:30001

Type text into the topic, such as Pandas are fabulous!.

  • Click Enter to separate between messages.

  • Click Ctrl + D to exit the produce command.

Consume (or read) the messages in the topic:

rpk topic consume panda-chat --brokers localhost:30001

Each message is shown with its metadata, like this:

{
"message": "Pandas are fabulous!\n",
"partition": 0,
"offset": 1,
"timestamp": "2022-02-10T15:52:35.251+02:00"
}

List the topics:

rpk topic list --brokers localhost:30001

Delete the cluster

Delete the cluster:

kind delete cluster

For more information, see the kind Deleting a cluster documentation.