Kubernetes Quickstart with Local Access on Kind
This guide will help you set up Redpanda locally for development and testing purposes on macOS, Windows, or Linux. Start here if you want to set up Redpanda quickly to try it out or for CI/CD purposes. If you want to use Amazon EKS or GKE, see the Kubernetes quickstart for cloud. If you want to use minikube, see the Kubernetes quickstart with local access on minikube.
This guide uses kind to create a local Kubernetes cluster in which Redpanda is deployed. It also verifies that the Kafka API port is exposed to your local network so that you can interact with Redpanda from your local machine and not just from inside the Kubernetes cluster.
To get up and running, you’ll create a cluster and deploy the Redpanda operator on the cluster. This guide will walk you through the steps to do that.
This quickstart guide walks you through the following steps:
-
Create a Kubernetes cluster
-
Install cert-manager
-
Install
rpk
-
Install the Redpanda operator
-
Install and connect to a Redpanda cluster
-
Check the
/etc/hosts
file -
Start streaming
-
Clean up
The steps in this guide limit the number of nodes to one because multi-node local clusters are not currently supported. For details on how to access Redpanda outside the Kubernetes network, see the Connecting remotely to Kubernetes guide.
In the steps below, the .yaml file that you use to install Redpanda sets developerMode: true . If you want to set developerMode: false , for optimal configuration it is recommended that you run rpk redpanda tune all directly on the host before you create a Redpanda cluster. You can find more information about the command as well as tuning recommendations in the Set Redpanda production mode documentation. If rpk is not available, verify that fs.aio-max-nr is set to 1048576 or greater. You can set fs.aio-max-nr by running sysctl -w fs.aio-max-nr=1048576 .
|
Prerequisites
Before you install Redpanda, verify that you have the following software installed on the machine where you want to run Redpanda:
-
kubectl version 1.19 or later
-
Helm version 3.0.0 or later
-
kind version 0.11.1 or later
-
Go v1.17 or later (only required for Windows setup)
-
jq - This setup uses jq to set the Redpanda
$VERSION
environment variable. The steps below assume that you have jq installed, but if you don’t, you can run the same commands by replacing the$VERSION
environment variable with the version of Redpanda that you’re using. Run the one of the commands below to install jq:
-
brew
-
apt
brew install jq
sudo apt-get update && \ sudo apt-get install jq
Step 1: Create a Kubernetes cluster
Use kind to create a Kubernetes cluster on your local machine.
kind is a tool that lets you create local Kubernetes clusters using Docker. After you install kind, verify that port 30001
is free on your machine before you set up a cluster or cluster creation will fail.
To enable access from your operating system to the cluster, create a kind-external.yaml
file that contains the following text:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30001
hostPort: 30001
Set up a cluster with this command:
kind create cluster --config kind-external.yaml
Step 2: Install cert-manager
The Redpanda operator requires cert-manager to create certificates for TLS communication. You can install cert-manager with Helm or kubectl.
Before you install cert-manager, verify that the version you’re installing is compatible with your Kubernetes version. The cert-manager Supported Releases documentation lists which Kubernetes versions are supported.
Use this command to install cert-manager with Helm:
helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.8.0 \
--set installCRDs=true
Verify the cert-manager installation
Use the verification procedure in the cert-manager documentation to verify that cert-manager is deployed correctly.
Step 3: Install rpk
Redpanda Keeper, or rpk
, is a CLI utility that you use to interact with Redpanda. Install rpk
on the host machine with this command:
brew install redpanda-data/tap/redpanda
On Windows, you must compile rpk yourself with Go. You can find the rpk source files in GitHub under redpanda/src/go/rpk.
|
Step 4: Use Helm to install the Redpanda operator
-
Run the following command to use Helm to add the Redpanda chart repository and update it:
helm repo add redpanda https://charts.vectorized.io/ && \ helm repo update
-
Use the following command to set the
$VERSION
environment variable to the latest operator and Redpanda version:export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)
If you prefer to manually enter the Redpanda version, or if you want to verify the version, run this command to return the installed version:
curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | grep tag_name
You can find information about the versions of the operator in the list of operator releases.
-
Install the Redpanda operator CRD with the bash or zsh command below:
-
bash
-
zsh
kubectl apply \ -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION
noglob kubectl apply \ -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION
-
-
Install the Redpanda operator on your Kubernetes cluster with this command:
helm install \ redpanda-operator \ redpanda/redpanda-operator \ --namespace redpanda-system \ --create-namespace \ --version $VERSION
Step 5: Install and connect to a Redpanda cluster
After you set up Redpanda in your Kubernetes cluster, you can use the sample configuration files in GitHub to install a cluster and see Redpanda in action.
The example here is an imaginary chat application, panda-chat
, but you can replace panda-chat
with any string. In this example, panda-chat
has five chat rooms.
Complete the following steps to manage a stream of events from panda-chat
:
-
Create a namespace for the cluster with this command:
kubectl create ns panda-chat
-
Install a single-node cluster like this (note that only single-node clusters are supported for local access clusters):
kubectl apply \ -n panda-chat \ -f https://raw.githubusercontent.com/redpanda-data/redpanda/dev/src/go/k8s/config/samples/one_node_external.yaml
You can view the resource configuration options, such as storage capacity, network configuration, or TLS configuration in the cluster_types file in GitHub. You can also find additional sample configuration files.
Step 6: Check the /etc/hosts
file
Map 0.local.rp
to 127.0.0.1
on the host machine that runs the kind cluster. The /etc/hosts
file will contain a line similar to this:
127.0.0.1 0.local.rp
The local.rp
address is specified in the one_node_external.yaml
file that you used in the previous step.
If you’re running Windows, this is the location of the file:
C:\Windows\System32\drivers\etc\hosts
Step 7: Do some streaming
The Redpanda image contains the rpk
and redpanda
binaries. rpk
is a CLI utility that you can use to work with your Redpanda nodes. See the rpk commands documentation for a full list of commands.
Follow these steps to start working with the panda-chat
cluster you created in the previous section.
-
Create a
panda-chat
topic with five partitions:rpk topic create panda-chat -p 5 --brokers localhost:30001
-
Produce messages to the topic:
rpk topic produce panda-chat --brokers localhost:30001
-
Type text into the topic, such as
Pandas are fabulous!
.-
Press Enter to separate between messages.
-
Press Ctrl + D to exit the produce command.
-
-
Consume (or read) the messages in the topic:
rpk topic consume panda-chat --brokers localhost:30001
Each message is shown with its metadata, like this:
{ "message": "Pandas are fabulous!\n", "partition": 0, "offset": 1, "timestamp": "2022-02-10T15:52:35.251+02:00" }
-
List the topics:
rpk topic list --brokers localhost:30001
Step 8: Clean up
Now that you’ve completed the quickstart, you can delete your cluster with the following command:
kind delete cluster
See the kind Deleting a cluster documentation for more information.