23.2 23.1 22.3 22.2 22.1 21.11

Kubernetes Quickstart for Cloud

This guide will help you set up Redpanda for development and testing purposes on any operating system. Start here if you want to set up Redpanda quickly to try it out or for CI/CD purposes.

The steps here install Redpanda on Amazon EKS, Google Kubernetes Engine (GKE), and DigitalOcean. If you want to run Redpanda using kind with local access, see the Kubernetes quickstart with local access on kind. If you want to use minikube, see the Kubernetes quickstart with local access on minikube.

To get up and running, you’ll create a cluster and deploy the Redpanda operator on the cluster. This guide will walk you through the steps to do that.

This quickstart guide walks you through the following steps:

  1. Create a Kubernetes cluster

  2. Configure the kubectl context

  3. Install cert-manager

  4. Install the Redpanda operator

  5. Install and connect to a Redpanda cluster

  6. Start streaming

  7. Clean up

Because the goal of this quickstart guide is to get you up and running as quickly as possible, you will run the client in the same cluster where you run Redpanda. This simplifies network connectivity. For details on how to access Redpanda outside the Kubernetes network, see the Connecting remotely to Kubernetes guide.

In the steps below, the .yaml file that you use to install Redpanda sets developerMode: true. If you want to set developerMode: false, for optimal configuration it is recommended that you run rpk redpanda tune all directly on the host before you create a Redpanda cluster. You can find more information about the command as well as tuning recommendations in the Set Redpanda production mode documentation. If rpk is not available, verify that fs.aio-max-nr is set to 1048576 or greater. You can set fs.aio-max-nr by running sysctl -w fs.aio-max-nr=1048576.

Prerequisites

Before you install Redpanda, verify that you have the following software installed on the machine where you want to run Redpanda:

  • kubectl version 1.19 or later

  • Helm version 3.0.0 or later

  • jq - This setup uses jq to set the Redpanda $VERSION environment variable. The steps below assume that you have jq installed, but if you don’t, you can run the same commands by replacing the $VERSION environment variable with the version of Redpanda that you’re using. Run the one of the commands below to install jq:

  • brew

  • apt

brew install jq
sudo apt-get update && \ sudo apt-get install jq

Amazon EKS prerequisites

In addition, if you’re using Amazon EKS, you will need the following prerequisites:

Step 1: Create a Kubernetes cluster

Select the cloud provider that you want to use to create your Kubernetes cluster:

  • GKE

  • Amazon EKS

  • DigitalOcean

Complete the Before You Begin steps in the Google Kubernetes Engine Quickstart. Then create a cluster with this command:

gcloud container clusters create redpanda --machine-type n1-standard-4 --num-nodes=1
You may need to add a --region or --zone to this command if defaults are not set. The Google Cloud documentation has information on how to set the defaults.

Use the Amazon EKS Getting Started Guide to set up Amazon EKS. When you finish, you’ll have eksctl installed so that you can create and delete clusters in Amazon EKS. Then, create an Amazon EKS cluster with:

eksctl create cluster \ --name redpanda \ --nodegroup-name standard-workers \ --node-type m5.xlarge \ --nodes 1 \ --nodes-min 1 \ --nodes-max 1

Alternatively, if you want to create an Amazon EKS cluster that is auto-tuned for running Redpanda, copy the following text and save it as eks-bootstrap.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: redpanda-eks
  region: us-west-1
nodeGroups:
  - name: standard-workers
    desiredCapacity: 1
    minSize: 1
    maxSize: 2
    instanceType: m5.xlarge
    preBootstrapCommands:
      - "curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | sudo -E bash && sudo yum -y install redpanda"
      - "sudo rpk mode prod && sudo rpk redpanda tune all"
      - "sudo yum -y remove redpanda"

The preBootstrapCommands in the file downloads the Redpanda RPM on the host, installs Redpanda, and runs the tuning. This sample file contains the following parameters that you might want to change:

  • instanceType: m5.xlarge

  • desiredCapacity: 1 - This sets the number of nodes that you want to create

Finally, create the Amazon EKS cluster with the following command:

eksctl create cluster -f eks-bootstrap.yaml

First, set up your DigitalOcean account and install doctl. Remember to set up your personal access token. You can find additional information in the DigitalOcean set up documentation. Then use the following command to create a cluster for your Redpanda deployment:

doctl kubernetes cluster create redpanda --wait --size s-4vcpu-8gb

Step 2: Configure the kubectl context

Most cloud utility tools will automatically change your kubectl configuration file.

To verify that you’re in the correct context, run the following command:

kubectl config current-context

On GKE for example, the output will look similar to this:

gke_myproject_us-west1_redpanda

If you’re running multiple clusters or if the configuration file wasn’t set up automatically, look for more information in the Kubernetes Configure Access to Multiple Clustersdocumentation.

Step 3: Install cert-manager

The Redpanda operator requires cert-manager to create certificates for TLS communication. You can install cert-manager with Helm or kubectl.

Use this command to install cert-manager with Helm:

helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.8.0 \
  --set installCRDs=true

Verify the cert-manager installation

You may have to wait a few minutes for cert-manager to be ready before you continue to the next step. Use the verification procedure in the cert-manager documentation to verify that cert-manager is deployed correctly.

Step 4: Use Helm to install the Redpanda operator

  1. Run the following command to use Helm to add the Redpanda chart repository and update it:

    helm repo add redpanda https://charts.vectorized.io/ && \
helm repo update

  2. Use the following command to set the $VERSION environment variable to the latest operator and Redpanda version:

    export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)

    If you prefer to manually enter the Redpanda version, or if you want to verify the version, run this command to return the installed version:

    curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | grep tag_name

    You can find information about the versions of the operator in the list of operator releases.

  3. Install the Redpanda operator CRD with the bash or zsh command below:

    • bash

    • zsh

    kubectl apply \ -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION
    noglob kubectl apply \ -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION

  4. Install the Redpanda operator on your Kubernetes cluster with this command:

    helm install \
  redpanda-operator \
  redpanda/redpanda-operator \
  --namespace redpanda-system \
  --create-namespace \
  --version $VERSION

Step 5: Install and connect to a Redpanda cluster

After you set up Redpanda in your Kubernetes cluster, you can use the sample configuration files in GitHub to install a cluster and see Redpanda in action.

The example here is an imaginary chat application, panda-chat, but you can replace panda-chat with any string. In this example, panda-chat has five chat rooms.

Complete the following steps to manage a stream of events from panda-chat:

  1. Create a namespace for the cluster with this command:

    kubectl create ns panda-chat

  2. Install a single-node cluster like this:

    kubectl apply \
-n panda-chat \
-f https://raw.githubusercontent.com/redpanda-data/redpanda/dev/src/go/k8s/config/samples/one_node_cluster.yaml

You can view the resource configuration options, such as storage capacity, network configuration, or TLS configuration in the cluster_types file in GitHub. You can also find additional sample configuration files.

Step 6: Do some streaming

The Redpanda image contains the rpk and redpanda binaries. Redpanda Keeper, or rpk, is a CLI utility that you can use to work with your Redpanda nodes. See the rpk commands documentation for a full list of commands.

Follow these steps to start working with the panda-chat cluster you created in the previous section.

  1. First, check the status of the cluster with this command:

    kubectl -n panda-chat run -ti --rm \
--restart=Never \
--image docker.redpanda.com/redpandadata/redpanda:$VERSION \
-- rpk --brokers one-node-cluster-0.one-node-cluster.panda-chat.svc.cluster.local:9092 \
cluster info

  2. Next, create a topic in the cluster. This command creates five chat rooms in the panda-chat cluster:

    kubectl -n panda-chat run -ti --rm \
--restart=Never \
--image docker.redpanda.com/redpandadata/redpanda:$VERSION \
-- rpk --brokers one-node-cluster-0.one-node-cluster.panda-chat.svc.cluster.local:9092 \
topic create chat-rooms -p 5

  3. Run this command to view the list of topics:

    kubectl -n panda-chat run -ti --rm \
--restart=Never \
--image docker.redpanda.com/redpandadata/redpanda:$VERSION \
-- rpk --brokers one-node-cluster-0.one-node-cluster.panda-chat.svc.cluster.local:9092 \
topic list

  4. Now you can produce to the topic with this command:

    kubectl -n panda-chat run -ti --rm \
--restart=Never \
--image docker.redpanda.com/redpandadata/redpanda \
-- rpk topic produce chat-rooms --brokers one-node-cluster-0.one-node-cluster.panda-chat.svc.cluster.local:9092

  5. Type text into the topic, such as Pandas are fabulous!.

    • Press Enter to separate between messages.

    • Press Ctrl + D to exit the produce command.

  6. Finally, consume, or read, from the topic with this command:

    kubectl -n panda-chat run -ti --rm \
--restart=Never \
--image docker.redpanda.com/redpandadata/redpanda \
-- rpk topic consume -n 1 chat-rooms --brokers one-node-cluster-0.one-node-cluster.panda-chat.svc.cluster.local:9092

In the rpk topic consume command above, -n 1 specifies the number of messages to print. If you produced more than one message in the previous step, you can change the number of messages to consume based on the number that you produced.

Step 7: Clean up

Now that you’ve completed the quickstart, you can use the following commands to delete your cluster:

  • GKE

  • EKS

  • DigitalOcean

gcloud container clusters delete redpanda

See the GKE Deleting a cluster documentation for more information.

eksctl delete cluster --name redpanda

See the Deleting an Amazon EKS cluster documentation for more information.

doctl kubernetes cluster delete

See the DigitalOcean cluster delete documentation for more information.