Deploy Redpanda in Kubernetes
This topic describes how to use the Redpanda Helm chart to deploy a Redpanda cluster in Kubernetes.
Make sure that your Kubernetes cluster meets the requirements.
Deploy Redpanda and Redpanda Console
In this step, you deploy Redpanda and Redpanda Console. Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. Redpanda Console is included as a subchart in the Redpanda Helm chart.
Add the Redpanda Helm chart repository and install cert-manager using Helm:
helm repo add redpanda https://charts.redpanda.com
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --set installCRDs=true --namespace cert-manager --create-namespace
The Redpanda Helm chart uses cert-manager to manage TLS certificates. TLS is enabled by default.
Install the Redpanda Helm chart:
helm install redpanda redpanda/redpanda \
--namespace redpanda \
Wait for the Redpanda cluster to be ready:
kubectl -n redpanda rollout status statefulset redpanda --watch
When the Redpanda cluster is ready, the output should look similar to the following:
statefulset rolling update complete 3 pods at revision redpanda-8654f645b4...
Verify that each Redpanda broker is scheduled on only one Kubernetes node:
kubectl get pod -n redpanda \
-o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -l \
Explore the default Kubernetes components
By default, the Redpanda Helm chart deploys the following Kubernetes components:
- A StatefulSet with three Pods.
- One PersistentVolumeClaim for each Pod, each with a capacity of 20Gi.
- A headless ClusterIP Service and a NodePort Service for each Kubernetes node that runs a Redpanda broker.
- Self-Signed TLS Certificates.
Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running.
kubectl get statefulset -n redpanda
NAME READY AGE
redpanda 3/3 3m11s
StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated.
Each Pod is also given a unique ordinal number in its name such as
A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number.
When a Pod in the StatefulSet is deleted and recreated,
it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced.
kubectl get pod -n redpanda
NAME READY STATUS RESTARTS AGE
redpanda-0 1/1 Running 0 6m9s
redpanda-1 1/1 Running 0 6m9s
redpanda-2 1/1 Running 0 6m9s
redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m
redpanda-configuration-smqv7 0/1 Completed 0 6m9s
redpanda-configuration Job updates the Redpanda runtime configuration.
Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using.
kubectl get persistentvolumeclaims -n redpanda
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s
datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s
datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s
The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients needs to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services:
- Internal using the Headless ClusterIP
- External using the NodePort
kubectl get service -n redpanda
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redpanda ClusterIP None <none> <none> 5m37s
redpanda-console ClusterIP 10.0.251.204 <none> 8080 5m
redpanda-external NodePort 10.96.137.220 <none> 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s
Headless ClusterIP Service
The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other.
An important requirement of distributed applications such as Redpanda is peer discovery:
The ability for each broker to find other brokers in the same cluster.
When each Pod is rolled out,
seed_servers field is updated with the FQDN of each Pod in the cluster so that they can discover each other.
kubectl -n redpanda exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml
External access is made available by a NodePort service that opens the following ports by default for the listeners:
|Node port||Pod port||Listener|
To learn more, see Networking and Connectivity in Kubernetes.
By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses cert-manager to generate two Certificate resources that provide Redpanda with self-signed certificates:
redpanda-default-certCertificate is the TLS certificate that is used by all listeners.
redpanda-default-root-certificateCertificate is the root certificate authority for the TLS certificates.
kubectl get certificate -n redpanda
NAME READY SECRET AGE
redpanda-default-cert True redpanda-default-cert 10m
redpanda-default-root-certificate True redpanda-default-root-certificate 10m
Customize the deployment
The Redpanda Helm chart is configured in the
To learn how to customize the Redpanda Helm chart, see Customize the Helm Chart.
Find the latest versions of the Redpanda Helm charts
helm install command always pulls the latest version of the chart. To list the current latest version of the Redpanda Helm chart, use the
helm search command:
helm search repo redpanda
NAME CHART VERSION APP VERSION DESCRIPTION
redpanda/redpanda 2.4.0 v22.3.9 Redpanda is the real-time engine for modern apps.
redpanda/console 0.3.3 v2.0.2 Helm chart to deploy Redpanda Console.
To find the versions that are installed on your machine, run the following:
helm list -n redpanda
If you're deploying Redpanda for production, see the production best practices.
See the Manage Kubernetes topics to learn how to customize the deployment to meet your needs.