Deploy a Redpanda Cluster in Google Kubernetes Engine
Deploy a secure Redpanda cluster and Redpanda Console in Google Kubernetes Engine (GKE). Then, use rpk
both as an internal client and an external client to interact with your Redpanda cluster from the command line. Your Redpanda cluster has the following security features:
-
SASL for authenticating users' connections.
-
TLS with self-signed certificates for secure communication between the cluster and clients.
Prerequisites
Before you begin, you must have the following:
-
Complete the 'Before you begin' steps and the 'Launch Cloud Shell' steps of the GKE quickstart. Cloud Shell comes preinstalled with the Google Cloud CLI, the
kubectl
command-line tool, and the Helm package manager. -
kubectl
. Minimum required Kubernetes version: 1.21.kubectl version --short --client
-
Helm. Minimum required Helm version: 3.6.0
helm version
Create a GKE cluster
Your GKE cluster must have one worker node available for each Redpanda broker that you plan to deploy in your Redpanda cluster. You also need to run the worker nodes on a machine type that supports the requirements and recommendations for production deployments.
In this step, you create a GKE cluster with three nodes on c2d-standard-8
machine types. Deploying three nodes allows your GKE cluster to support a Redpanda cluster with three brokers. The c2d-standard-8
instance type comes with:
-
2 cores per worker node, which is a requirement for production.
-
Local NVMe disks, which is recommended for best performance.
The Helm chart configures podAntiAffinity rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node.
|
Create a GKE cluster. Replace the <region>
placeholder with your own region.
gcloud container clusters create <cluster-name> \
--machine-type c2d-standard-8 \
--num-nodes=3 \
--local-nvme-ssd-block count=2
--region=<region>
To see all options that you can specify when creating a cluster, see the Cloud SDK reference. Or, for help creating a GKE cluster, see the GKE documentation. |
Create a StorageClass for your local NVMe disks
When you provisioned the Kubernetes cluster, you selected an instance type that comes with local NVMe disks. However, these disks are not automatically mounted or formatted upon creation. To use these local NVMe disks, you must mount and format them, and you must create the necessary PersistentVolumes (PVs). To automate this process, you can use a Container Storage Interface (CSI) driver.
In this step, you install the recommended local volume manager (LVM) CSI driver. Then, you create a StorageClass that references the LVM CSI driver and specifies the recommended XFS file system.
-
Install the LVM CSI driver:
helm repo add metal-stack https://helm.metal-stack.io helm repo update helm install csi-driver-lvm metal-stack/csi-driver-lvm \ --namespace csi-driver-lvm \ --create-namespace \ --set lvm.devicePattern='/dev/nvme[0-9]n[0-9]'
The
lvm.devicePattern
property specifies the pattern that the CSI driver uses to identify available NVMe volumes on your worker nodes. -
Create the StorageClass:
csi-driver-lvm-striped-xfs.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-driver-lvm-striped-xfs provisioner: lvm.csi.metal-stack.io reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: type: "striped" csi.storage.k8s.io/fstype: xfs
-
provisioner
: The LVM CSI driver responsible for provisioning the volume. -
reclaimPolicy
: TheRetain
policy ensures that the underlying volume is not deleted when the corresponding PVC is deleted. -
volumeBindingMode
: TheWaitForFirstConsumer
mode delays the binding and provisioning of a PersistentVolume until a Pod that uses the PVC is created. This mode is important for ensuring that the PV is created on the same node where the Pod will run because the PV will use the node’s local NVMe volumes. -
allowVolumeExpansion
: Allows the volume to be expanded after it has been provisioned. -
parameters.type
: Combines multiple physical volumes to create a single logical volume. In a striped setup, data is spread across the physical volumes in a way that distributes the I/O load evenly, improving performance by allowing parallel disk I/O operations. -
parameters.csi.storage.k8s.io/fstype
: Formats the volumes with the XFS file system. Redpanda Data recommends XFS for its enhanced performance with Redpanda workloads.
-
-
Apply the StorageClass:
kubectl apply -f csi-driver-lvm-striped-xfs.yaml
After applying this StorageClass, any PVC that references it will attempt to provision storage using the LVM CSI driver and the provided parameters.
Configure external access
Add inbound firewall rules to your instances so that external traffic can reach the following node ports on all Kubernetes worker nodes in the cluster:
-
31644
-
31092
-
30082
-
30081
For help creating firewall rules, see the Google VPC documentation.
Deploy Redpanda and Redpanda Console
In this step, you deploy Redpanda with SASL authentication and self-signed TLS certificates. Redpanda Console is included as a subchart in the Redpanda Helm chart.
-
Helm + Operator
-
Helm
-
Make sure that you have permission to install custom resource definitions (CRDs):
kubectl auth can-i create CustomResourceDefinition --all-namespaces
You should see
yes
in the output.You need these cluster-level permissions to install cert-manager and Redpanda Operator CRDs in the next steps.
-
Install cert-manager using Helm:
helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set installCRDs=true \ --namespace cert-manager \ --create-namespace
The Redpanda Helm chart uses cert-manager to enable TLS and manage TLS certificates by default.
-
Install the Redpanda Operator custom resource definitions (CRDs):
kubectl kustomize https://github.com/redpanda-data/redpanda-operator//src/go/k8s/config/crd | kubectl apply -f -
-
Deploy the Redpanda Operator:
helm repo add redpanda https://charts.redpanda.com helm upgrade --install redpanda-controller redpanda/operator \ --namespace <namespace> \ --set image.repository=docker.redpanda.com/redpandadata/redpanda-operator \ --create-namespace \ --timeout 1h
-
Ensure that the Deployment is successfully rolled out:
kubectl --namespace <namespace> rollout status --watch deployment/redpanda-controller-operator
deployment "redpanda-controller-operator" successfully rolled out
-
Install a Redpanda custom resource in the same namespace as the Redpanda Operator:
redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha1 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: external: domain: customredpandadomain.local auth: sasl: enabled: true users: - name: superuser password: secretpassword storage: persistentVolume: enabled: true storageClass: csi-driver-lvm-striped-xfs
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
-
external.domain
: The custom domain that each broker will advertise to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. -
auth.sasl.name
: Creates a superuser calledsuperuser
that can grant permissions to new users in your cluster using access control lists (ACLs). -
storage.persistentVolume.storageClass
: Points each PVC associated with the Redpanda brokers to thecsi-driver-lvm-striped-xfs
StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker.
-
-
Wait for the Redpanda Operator to deploy Redpanda using the Helm chart:
kubectl get redpanda --namespace <namespace> --watch
NAME READY STATUS redpanda True Redpanda reconciliation succeeded
This step may take a few minutes. You can watch for new Pods to make sure that the deployment is progressing:
kubectl get pod --namespace <namespace>
If it’s taking too long, see Troubleshoot.
-
Install cert-manager using Helm:
helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set installCRDs=true \ --namespace cert-manager \ --create-namespace
TLS is enabled by default. The Redpanda Helm chart uses cert-manager to manage TLS certificates by default.
-
Install Redpanda with SASL enabled:
helm repo add redpanda https://charts.redpanda.com export DOMAIN=customredpandadomain.local && \ helm install redpanda redpanda/redpanda \ --namespace <namespace> --create-namespace \ --set auth.sasl.enabled=true \ --set "auth.sasl.users[0].name=superuser" \ --set "auth.sasl.users[0].password=secretpassword" \ --set external.domain=${DOMAIN} \ --set "storage.persistentVolume.storageClass=csi-driver-lvm-striped-xfs" \ --wait \ --timeout 1h
-
external.domain
: The custom domain that each broker advertises to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. -
auth.sasl.name
: Creates a superuser calledsuperuser
that can grant permissions to new users in your cluster using access control lists (ACLs). -
storage.persistentVolume.storageClass
: Points each PVC associated with the Redpanda brokers to thecsi-driver-lvm-striped-xfs
StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker.
-
The installation displays some tips for getting started.
If the installation is taking a long time, see Troubleshoot.
Verify the deployment
When the Redpanda Helm chart is deployed, you should have:
-
Three Redpanda brokers. Each Redpanda broker runs inside a separate Pod and is scheduled on a separate worker node.
-
One PVC bound to a PV for each Redpanda broker. These PVs are what the Redpanda brokers use to store the Redpanda data directory with all your topics and metadata.
-
Verify that each Redpanda broker is scheduled on only one Kubernetes node:
kubectl get pod --namespace <namespace> \ -o=custom-columns=NODE:.spec.nodeName,POD_NAME:.metadata.name -l \ app.kubernetes.io/component=redpanda-statefulset
Example output:
NODE POD_NAME example-worker3 redpanda-0 example-worker2 redpanda-1 example-worker redpanda-2
-
Verify that each Redpanda broker has a bound PVC:
kubectl get persistentvolumeclaim \ --namespace <namespace> \ -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,STORAGECLASS:.spec.storageClassName
Example output:
NAME STATUS STORAGECLASS datadir-redpanda-0 Bound csi-driver-lvm-striped-xfs datadir-redpanda-1 Bound csi-driver-lvm-striped-xfs datadir-redpanda-2 Bound csi-driver-lvm-striped-xfs
Create a user
In this step, you use rpk
to create a new user. Then, you authenticate to Redpanda with the superuser to grant permissions to the new user. You’ll authenticate to Redpanda with this new user to create a topic in the next steps.
As a security best practice, you should use the superuser only to grant permissions to new users through ACLs. Never delete the superuser. You need the superuser to grant permissions to new users. |
-
Create a new user called
redpanda-twitch-account
with the passwordchangethispassword
:kubectl --namespace <namespace> exec -ti redpanda-0 -c redpanda -- \ rpk acl user create redpanda-twitch-account \ -p changethispassword
Example output:
Created user "redpanda-twitch-account".
-
Use the superuser to grant the
redpanda-twitch-account
user permission to execute all operations only for a topic calledtwitch-chat
.kubectl exec --namespace <namespace> -c redpanda redpanda-0 -- \ rpk acl create --allow-principal User:redpanda-twitch-account \ --operation all \ --topic twitch-chat \ -X user=superuser -X pass=secretpassword -X sasl.mechanism=SCRAM-SHA-512
Example output:
PRINCIPAL RESOURCE-TYPE RESOURCE-NAME OPERATION PERMISSION User:redpanda TOPIC twitch-chat ALL ALLOW
Start streaming
In this step, you authenticate to Redpanda with the redpanda-twitch-account
user to create a topic called twitch-chat
. This topic is the only one that the redpanda-twitch-account
user has permission to access. Then, you produce messages to the topic, and consume messages from it.
-
Create an alias to simplify the
rpk
commands:alias internal-rpk="kubectl --namespace <namespace> exec -i -t redpanda-0 -c redpanda -- rpk -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256"
-
Create a topic called
twitch-chat
:-
Helm + Operator
-
Helm
-
Create a Topic resource:
topic.yaml
apiVersion: cluster.redpanda.com/v1alpha1 kind: Topic metadata: name: twitch-chat spec: kafkaApiSpec: brokers: - "redpanda-0.redpanda.<namespace>.svc.cluster.local:9093" - "redpanda-1.redpanda.<namespace>.svc.cluster.local:9093" - "redpanda-2.redpanda.<namespace>.svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt"
-
Apply the Topic resource in the same namespace as your Redpanda cluster:
kubectl apply -f topic.yaml --namespace <namespace>
-
Check the logs of the Redpanda Operator to confirm that the topic was created:
kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace <namespace>
You should see that the Redpanda Operator reconciled the Topic resource. For example:
Example output
{ "level":"info", "ts":"2023-09-25T16:20:09.538Z", "logger":"TopicReconciler.Reconcile", "msg":"Starting reconcile loop", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"<namespace>" }, "namespace":"<namespace>", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432" } { "level":"info", "ts":"2023-09-25T16:20:09.581Z", "logger":"TopicReconciler.Reconcile", "msg":"reconciliation finished in 43.436125ms, next run in 3s", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"<namespace>" }, "namespace":"<namespace>", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432", "result": { "Requeue":false, "RequeueAfter":3000000000 } }
internal-rpk topic create twitch-chat
Example output:
TOPIC STATUS twitch-chat OK
-
-
Describe the topic:
internal-rpk topic describe twitch-chat
Expected output:
SUMMARY ======= NAME twitch-chat PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG partition_count 1 DYNAMIC_TOPIC_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG replication_factor 1 DYNAMIC_TOPIC_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG
-
Produce a message to the topic:
internal-rpk topic produce twitch-chat
-
Type a message, then press Enter:
Pandas are fabulous!
Example output:
Produced to partition 0 at offset 0 with timestamp 1663282629789.
-
Press Ctrl+C to finish producing messages to the topic.
-
Consume one message from the topic:
internal-rpk topic consume twitch-chat --num 1
Expected output:
{ "topic": "twitch-chat", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 }
Explore your topic in Redpanda Console
Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications.
In this step, you use port-forwarding to access Redpanda Console on your local network.
Because you’re using the Community Edition of Redpanda Console, you should not expose Redpanda Console outside your local network. The Community Edition of Redpanda Console does not provide authentication, and it connects to the Redpanda cluster as superuser. To use the Enterprise Edition, you need a license key, see Redpanda Licensing. |
-
Expose Redpanda Console to your localhost:
kubectl --namespace <namespace> port-forward svc/redpanda-console 8080:8080
The
kubectl port-forward
command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window. -
Open Redpanda Console on http://localhost:8080.
All your Redpanda brokers are listed along with their IP addresses and IDs.
-
Go to Topics > twitch-chat.
The message that you produced to the topic is displayed along with some other details about the topic.
-
Press Ctrl+C in the command-line to stop the port-forwarding process.
Configure external access to Redpanda
If you want to connect to the Redpanda cluster with external clients, Redpanda brokers must advertise an externally accessible address that external clients can connect to. External clients are common in Internet of Things (IoT) environments, or if you use external services that do not implement VPC peering in your network.
When you created the cluster, you set the external.domain
configuration to customredpandadomain.local
, which means that your Redpanda brokers are advertising the following addresses:
-
redpanda-0.customredpandadomain.local
-
redpanda-1.customredpandadomain.local
-
redpanda-2.customredpandadomain.local
To access your Redpanda brokers externally, you can map your worker nodes' IP addresses to these domains.
IP addresses can change. If the IP addresses of your worker nodes change, you must update your In a production environment, it’s a best practice to use ExternalDNS to manage DNS records for your brokers. See Use ExternalDNS for external access. |
-
Add mappings in your
/etc/hosts
file between your worker nodes' IP addresses and their custom domain names:sudo true && kubectl --namespace <namespace> get endpoints,node -A -o go-template='{{ range $_ := .items }}{{ if and (eq .kind "Endpoints") (eq .metadata.name "redpanda-external") }}{{ range $_ := (index .subsets 0).addresses }}{{ $nodeName := .nodeName }}{{ $podName := .targetRef.name }}{{ range $node := $.items }}{{ if and (eq .kind "Node") (eq .metadata.name $nodeName) }}{{ range $_ := .status.addresses }}{{ if eq .type "ExternalIP" }}{{ .address }} {{ $podName }}.${DOMAIN}{{ "\n" }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}' | envsubst | sudo tee -a /etc/hosts
/etc/hosts
203.0.113.3 redpanda-0.customredpandadomain.local 203.0.113.5 redpanda-1.customredpandadomain.local 203.0.113.7 redpanda-2.customredpandadomain.local
-
Save the root certificate authority (CA) to your local file system outside Kubernetes:
kubectl --namespace <namespace> get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt
-
Install
rpk
on your local machine, not on a Pod:-
Linux
-
macOS
-
Download the
rpk
archive for Linux, and make sure the version matches your Redpanda version.-
To download the latest version of
rpk
:curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
-
To download a version other than the latest:
curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-linux-amd64.zip
-
-
Ensure that you have the folder
~/.local/bin
:mkdir -p ~/.local/bin
-
Add it to your
$PATH
:export PATH="~/.local/bin:$PATH"
-
Unzip the
rpk
files to your~/.local/bin/
directory:unzip rpk-linux-amd64.zip -d ~/.local/bin/
-
Run
rpk version
to display the version of the rpk binary:rpk version
23.2.17 (rev ae9a71b)
-
If you don’t have Homebrew installed, install it.
-
Install
rpk
:brew install redpanda-data/tap/redpanda
-
Run
rpk version
to display the version of the rpk binary:rpk version
23.2.17 (rev ae9a71b)
This method installs the latest version of rpk
, which is supported only with the latest version of Redpanda.
-
-
Configure
rpk
to connect to your cluster using the pre-configured profile:rpk profile create --from-profile <(kubectl get configmap --namespace <namespace> redpanda-rpk -o go-template='{{ .data.profile }}') <profile-name>
Replace
<profile-name>
with the name that you want to give thisrpk
profile. -
Test the connection:
rpk cluster info -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256
Explore the default Kubernetes components
By default, the Redpanda Helm chart deploys the following Kubernetes components:
-
A StatefulSet with three Pods.
-
One PersistentVolumeClaim for each Pod, each with a capacity of 20Gi.
-
A headless ClusterIP Service and a NodePort Service for each Kubernetes node that runs a Redpanda broker.
StatefulSet
Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running.
kubectl get statefulset --namespace <namespace>
Example output:
NAME READY AGE redpanda 3/3 3m11s
StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated.
Each Pod is also given a unique ordinal number in its name such as redpanda-0
.
A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number.
When a Pod in the StatefulSet is deleted and recreated,
it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced.
kubectl get pod --namespace <namespace>
Expected output:
NAME READY STATUS RESTARTS AGE
redpanda-0 1/1 Running 0 6m9s
redpanda-1 1/1 Running 0 6m9s
redpanda-2 1/1 Running 0 6m9s
redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m
redpanda-configuration-smqv7 0/1 Completed 0 6m9s
The redpanda-configuration job updates the Redpanda runtime configuration.
|
PersistentVolumeClaim
Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using.
kubectl get persistentvolumeclaims --namespace <namespace>
Expected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s
datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s
datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s
Service
The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients need to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services:
-
Internal using the Headless ClusterIP
-
External using the NodePort
kubectl get service --namespace <namespace>
Expected output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redpanda ClusterIP None <none> <none> 5m37s
redpanda-console ClusterIP 10.0.251.204 <none> 8080 5m
redpanda-external NodePort 10.96.137.220 <none> 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s
Headless ClusterIP Service
The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other.
An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster.
When each Pod is rolled out, its seed_servers
field is updated with the FQDN of each Pod in the cluster so that they can discover each other.
kubectl --namespace <namespace> exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml
redpanda:
data_directory: /var/lib/redpanda/data
empty_seed_starts_cluster: false
seed_servers:
- host:
address: redpanda-0.redpanda.<namespace>.svc.cluster.local.
port: 33145
- host:
address: redpanda-1.redpanda.<namespace>.svc.cluster.local.
port: 33145
- host:
address: redpanda-2.redpanda.<namespace>.svc.cluster.local.
port: 33145
NodePort Service
External access is made available by a NodePort service that opens the following ports by default:
Listener | Node Port | Container Port |
---|---|---|
Schema Registry |
30081 |
8081 |
HTTP Proxy |
30082 |
8083 |
Kafka API |
31092 |
9094 |
Admin API |
31644 |
9644 |
To learn more, see Networking and Connectivity in Kubernetes.
TLS Certificates
By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses cert-manager to generate four Certificate resources that provide Redpanda with self-signed certificates for internal and external connections.
Having separate certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications.
kubectl get certificate --namespace <namespace>
NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True
-
redpanda-default-cert
: Self-signed certificate for internal communications. -
redpanda-default-root-certificate
: Root certificate authority for the internal certificate. -
redpanda-external-cert
: Self-signed certificate for external communications. -
redpanda-external-root-certificate
: Root certificate authority for the external certificate.
By default, all listeners are configured with the same certificate. To configure separate TLS certificates for different listeners, see TLS for Redpanda in Kubernetes.
The Redpanda Helm chart provides self-signed certificates for convenience. In a production environment, it’s best to use certificates from a trusted Certificate Authority (CA) or integrate with your existing CA infrastructure. |
Troubleshoot
Before troubleshooting your cluster, make sure that you have all the prerequisites.
HelmRelease is not ready
If you are using the Redpanda Operator with Helm, you may see the following message while waiting for a Redpanda custom resource to be deployed:
NAME READY STATUS
redpanda False HelmRepository 'redpanda/redpanda-repository' is not ready
redpanda False HelmRelease 'redpanda/redpanda' is not ready
While the deployment process can sometimes take a few minutes, a prolonged 'not ready' status may indicate an issue. Follow the steps below to investigate:
-
Check the status of the HelmRelease:
kubectl describe helmrelease <redpanda-resource-name> --namespace <namespace>
-
Review the Redpanda Operator logs:
kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace <namespace>
StatefulSet never rolls out
If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available.
To identify the Pods that are pending, use the following command:
kubectl get pod --namespace <namespace>
The response includes a list of Pods in the StatefulSet and their status.
To view logs for a specific Pod, use the following command.
kubectl logs -f <pod-name> --namespace <namespace>
You can use the output to debug your deployment.
Unable to mount volume
If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data.
-
If you’re using StorageClasses with dynamic provisioners (default), ensure they exist:
kubectl get storageclass
-
If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in
storage.persistentVolume.size
:kubectl get persistentvolume --namespace <namespace>
To learn how to configure different storage volumes, see Configure Storage.
Dig not defined
This error means that you are using an unsupported version of Helm:
Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined
Ensure that you are using the minimum required version: 3.6.0.
helm version
Repository name already exists
If you see this error, remove the redpanda
chart repository, then try installing it again.
helm repo remove redpanda
helm repo add redpanda https://charts.redpanda.com
helm repo update
Fatal error during checker "Data directory is writable" execution
This error appears when Redpanda does not have write access to your configured storage volume under storage
in the Helm chart.
Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test_file: permission denied
To fix this error, set statefulset.initContainers.setDataDirOwnership.enabled
to true
so that the initContainer can set the correct permissions on the data directories.
Cannot patch "redpanda" with kind StatefulSet
This error appears when you run helm upgrade
with the --values
flag but do not include all your previous overrides.
Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
Ensure to do one of the following:
-
Include all the value overrides from the previous installation or upgrade using either the
--set
or the--values
flags. -
Use the
--reuse-values
flag.Do not use the --reuse-values
flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart.
Cannot patch "redpanda-console" with kind Deployment
This error appears if you try to upgrade your deployment and you already have console.enabled
set to true
.
Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
To fix this error, set console.enabled
to false
so that Helm doesn’t try to deploy Redpanda Console again.
For more troubleshooting steps, see Troubleshoot Redpanda in Kubernetes.
Next steps
When you’re ready to use a registered domain, make sure to remove your entries from the /etc/hosts file, and see Configure External Access through a NodePort Service.
|