23.2 23.1 22.3 22.2 22.1 21.11

Deploy Redpanda in Kubernetes

This topic describes how to use the Redpanda Helm chart to deploy a Redpanda cluster in Kubernetes.

Prerequisites

Make sure that your Kubernetes cluster meets the requirements.

Deploy Redpanda and Redpanda Console

In this step, you deploy Redpanda and Redpanda Console. Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. Redpanda Console is included as a subchart in the Redpanda Helm chart.

  1. Install cert-manager using Helm:

    helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --set installCRDs=true \
  --namespace cert-manager  \
  --create-namespace

    TLS is enabled by default. The Redpanda Helm chart uses cert-manager to manage TLS certificates by default.

  2. Install the Redpanda Helm chart:

    helm install redpanda redpanda/redpanda \
  --namespace redpanda \
  --create-namespace

  3. Wait for the Redpanda cluster to be ready:

    kubectl -n redpanda rollout status statefulset redpanda --watch

    When the Redpanda cluster is ready, the output should look similar to the following:

    statefulset rolling update complete 3 pods at revision redpanda-8654f645b4...

  4. Verify that each Redpanda broker is scheduled on only one Kubernetes node:

    kubectl get pod -n redpanda  \
-o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -l \
app.kubernetes.io/component=redpanda-statefulset

    Example output:

    example-worker3   redpanda-0
example-worker2   redpanda-1
example-worker    redpanda-2

Explore the default Kubernetes components

By default, the Redpanda Helm chart deploys the following Kubernetes components:

StatefulSet

Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running.

kubectl get statefulset -n redpanda

Example output:

NAME       READY   AGE
redpanda   3/3     3m11s

StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated. Each Pod is also given a unique ordinal number in its name such as redpanda-0. A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number. When a Pod in the StatefulSet is deleted and recreated, it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced.

kubectl get pod -n redpanda

Example output:

NAME                              READY   STATUS      RESTARTS        AGE
redpanda-0                        1/1     Running     0               6m9s
redpanda-1                        1/1     Running     0               6m9s
redpanda-2                        1/1     Running     0               6m9s
redpanda-console-5ff45cdb9b-6z2vs 1/1     Running     0               5m
redpanda-configuration-smqv7      0/1     Completed   0               6m9s
The redpanda-configuration Job updates the Redpanda runtime configuration.

PersistentVolumeClaim

Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using.

kubectl get persistentvolumeclaims -n redpanda

Example output:

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-redpanda-0   Bound    pvc-3311ade3-de84-4027-80c6-3d8347302962   20Gi       RWO            standard       75s
datadir-redpanda-1   Bound    pvc-4ea8bc03-89a6-41e4-b985-99f074995f08   20Gi       RWO            standard       75s
datadir-redpanda-2   Bound    pvc-45c3555f-43bc-48c2-b209-c284c8091c45   20Gi       RWO            standard       75s

Service

The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients needs to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services:

kubectl get service -n redpanda

Example output:

NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                       AGE
redpanda            ClusterIP   None            <none>        <none>                                                        5m37s
redpanda-console    ClusterIP   10.0.251.204    <none>        8080                                                          5m
redpanda-external   NodePort    10.96.137.220   <none>        9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP   5m37s

Headless ClusterIP Service

The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other.

An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster. When each Pod is rolled out, its seed_servers field is updated with the FQDN of each Pod in the cluster so that they can discover each other.

kubectl -n redpanda exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml
redpanda:
  data_directory: /var/lib/redpanda/data
  empty_seed_starts_cluster: false
  seed_servers:
  - host:
      address: redpanda-0.redpanda.redpanda.svc.cluster.local.
      port: 33145
  - host:
      address: redpanda-1.redpanda.redpanda.svc.cluster.local.
      port: 33145
  - host:
      address: redpanda-2.redpanda.redpanda.svc.cluster.local.
      port: 33145

NodePort Service

External access is made available by a NodePort service that opens the following ports by default for the listeners:

Node port Pod port Listener

30081

8081

Schema Registry

30082

8083

HTTP Proxy

31092

9094

Kafka API

31644

9644

Admin API

To learn more, see Networking and Connectivity in Kubernetes.

TLS Certificates

By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses cert-manager to generate two Certificate resources that provide Redpanda with self-signed certificates:

  • The redpanda-default-cert Certificate is the TLS certificate that is used by all listeners.

  • The redpanda-default-root-certificate Certificate is the root certificate authority for the TLS certificates.

kubectl get certificate -n redpanda
NAME                                 READY   SECRET                               AGE
redpanda-default-cert                True    redpanda-default-cert                10m
redpanda-default-root-certificate    True    redpanda-default-root-certificate    10m

Customize the deployment

The Redpanda Helm chart is configured in the values.yaml file.

To learn how to customize the Redpanda Helm chart, see Customize the Helm Chart.

Find the latest versions of the Redpanda Helm charts

The helm install command always pulls the latest version of the chart. To list the current latest version of the Redpanda Helm chart, use the helm search command:

helm search repo redpanda

Example output:

NAME             	CHART VERSION	APP VERSION	DESCRIPTION
redpanda/redpanda	2.4.0        	v22.3.9    	Redpanda is the real-time engine for modern apps.
redpanda/console 	0.3.3        	v2.0.2     	Helm chart to deploy Redpanda Console.

To find the versions that are installed on your machine, run the following:

helm list -n redpanda

Production considerations

If you’re deploying Redpanda for production, see the production best practices.

Next steps

See the Manage Kubernetes topics to learn how to customize the deployment to meet your needs.