Migrate from the Redpanda Helm chart
If you are using the Redpanda Helm chart, you can migrate to the Redpanda Operator and use it to manage your Helm deployment.
The Redpanda Operator extends Kubernetes with custom resource definitions (CRDs), which allow Redpanda clusters to be treated as native Kubernetes resources. The primary resource that the Redpanda Operator uses to represent a Redpanda cluster is the Redpanda resource.
Here is an example of a Redpanda custom resource:
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
name: <cluster-name>
spec:
clusterSpec:
image:
tag: <redpanda-version>
-
metadata.name: Name to assign the Redpanda cluster. This name is also assigned to the Helm release. -
spec.clusterSpec.image.tag: The version of Redpanda to deploy (for example,v25.2.1). Starting from operator v25.1.1, the operator bundles Helm charts internally and automatically selects the appropriate chart version. -
spec.clusterSpec: This is where you can configure the Redpanda CRD with your values overrides from the Redpanda Helm chart.
Supported migration paths
The following table summarizes which Helm chart versions you can migrate from and which Redpanda Operator versions to install.
| Helm Chart Version | Operator Version | Notes |
|---|---|---|
|
|
You must first |
|
|
After installing or upgrading to Helm chart 5.9.x or 5.10.x, you can install the Redpanda Operator v2.4.x. |
Prerequisites
Before migrating to the Redpanda Operator, you must have:
-
The name of your existing Helm release and the latest version of the Redpanda Helm chart that you have deployed.
helm list -AIn this example the chart version is 5.9.1 and the release name is
redpanda.NAME CHART redpanda redpanda-5.9.1
Make a note of your name and version for the next steps. You’ll need to configure your Redpanda custom resource with these details.
-
Your values overrides.
helm get values <cluster-name> --namespace <namespace>You should see your overrides in YAML format. You’ll need to configure your Redpanda custom resource with these details.
| Before implementing any changes in your production environment, Redpanda Data recommends testing the migration in a non-production environment. |
Rolling restart during migration
When you apply the Redpanda custom resource, the operator triggers a rolling restart of all broker pods. The rolling restart is unavoidable during migration, regardless of how closely your clusterSpec values match the existing Helm values.
How the rolling restart works
The migration triggers the following sequence:
-
Resource adoption: The operator uses server-side apply to take ownership of the existing Helm-managed StatefulSet. It patches the StatefulSet with new labels (generation, config version, and ownership) and sets the Redpanda custom resource as the owner reference.
-
StatefulSet spec update: The operator re-renders the StatefulSet from its own templates. Any difference, even metadata or label changes, creates a new
ControllerRevision. -
Pod rolling: The operator compares each pod’s
StatefulSetRevisionLabelagainst the latestControllerRevision. Because the existing pods were created under the old Helm-managed revision, the operator flags every pod for rolling. -
One-at-a-time deletion: The operator deletes pods one at a time:
-
Checks cluster health through the admin API.
-
Deletes the pod if the cluster is healthy, then requeues after 10 seconds.
-
Waits for the cluster to stabilize before rolling the next pod.
-
Skips deletion and requeues if the cluster is unhealthy.
-
Impact by cluster configuration
| Scenario | Impact |
|---|---|
3+ brokers, replication factor (RF) ≥ 3 |
No data loss and no downtime for consumers or producers configured with |
3 brokers, RF = 1 |
Partitions on the restarting broker are unavailable for the duration of that broker’s restart. |
Single broker |
Full outage for the duration of the restart. |
Recommended producer settings
To avoid message loss during the rolling restart, configure producers with the following settings:
-
acks=all(or-1): Ensures the Raft quorum (majority of replicas) commits the write before acknowledging it. -
retries: HandlesNOT_LEADER_FOR_PARTITIONerrors during leader elections. Set this to a high value. -
enable.idempotence=true: Prevents duplicate messages from retries.
Migrate to the Redpanda Operator and Helm
To migrate to the latest Redpanda Operator and use it to manage your Helm deployment, follow these steps.
-
Make sure that you have permission to install custom resource definitions (CRDs):
kubectl auth can-i create CustomResourceDefinition --all-namespacesYou should see
yesin the output.You need these cluster-level permissions to install the Redpanda Operator CRDs in the next steps.
-
Install the Redpanda Operator. Starting in v25.2, the Redpanda Operator can manage Redpanda clusters in any namespace:
-
To deploy in cluster scope, use:
helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade --install redpanda-controller redpanda/operator \ --namespace <namespace> \ --create-namespace \ --version v26.1.4 \ (1) --set crds.enabled=true (2)1 This flag specifies the exact version of the Redpanda Operator Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. 2 This flag ensures that the CRDs are installed as part of the Redpanda Operator deployment. This command deploys the Redpanda Operator in cluster scope (default in v25.2+), allowing it to manage Redpanda clusters across multiple namespaces.
-
To deploy in namespace scope (managing only resources within its deployment namespace), use:
helm upgrade --install redpanda-controller redpanda/operator \ --namespace <namespace> \ --create-namespace \ --version v26.1.4 \ --set crds.enabled=true \ --set 'additionalCmdFlags=["--namespace=<cluster-namespace>"]' (1)1 This flag restricts the Redpanda Operator to manage resources only within the specified namespace.
-
-
Ensure that the Deployment is successfully rolled out:
kubectl --namespace <namespace> rollout status -w deployment/redpanda-controller-operatordeployment "redpanda-controller" successfully rolled out
-
Configure a Redpanda custom resource that Redpanda Operator will use to adopt your Redpanda cluster.
Replace the placeholders with the values identified in the Prerequisites.
redpanda-cluster.yamlapiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: annotations: cluster.redpanda.com/managed: "true" creationTimestamp: null name: <cluster-name> (1) spec: clusterSpec: image: tag: <redpanda-version> (2) <chart-overrides> (3)1 Replace with your Helm release name. 2 Specify the Redpanda version to deploy (for example, v25.2.1). The operator automatically uses the appropriate Helm chart version for the specified Redpanda version.-
Starting from operator v25.1.1, the
chartReffields (includingchartRef.chartVersion) are deprecated. The operator bundles Helm charts internally. -
If your existing Helm deployment is on an older version, verify compatibility in Kubernetes Compatibility before migrating.
3 Replace with your chart overrides. For details on the structure and configuration options of the Redpanda custom resource, refer to the Redpanda Operator CRD reference. -
-
Adopt the Redpanda cluster by creating an instance of the Redpanda custom resource:
kubectl apply -f redpanda-cluster.yaml --namespace <namespace> -
Wait for the Redpanda resource to successfully reach a
deployedstate:kubectl get redpanda --namespace <namespace> --watchExample output:
NAME READY STATUS redpanda True Redpanda reconciliation succeeded
Roll back from Redpanda Operator to Helm
If you migrated to the Redpanda Operator and want to revert to using only Helm, follow these steps to uninstall the Redpanda Operator:
Follow the steps in exact order to avoid race conditions between the Redpanda Operator’s reconciliation loop and Kubernetes garbage collection.
-
Delete all Redpanda-related custom resources:
kubectl delete users --namespace <namespace> --all kubectl delete topics --namespace <namespace> --all kubectl delete schemas --namespace <namespace> --all kubectl delete redpanda --namespace <namespace> --all -
Make sure requests for those resources return no results. For example, if you had a Redpanda cluster named
redpandain the namespace<namespace>, run:kubectl get redpanda --namespace <namespace> -
Uninstall the Redpanda Operator Helm release:
helm uninstall redpanda-controller --namespace <namespace>Helm does not uninstall CRDs by default when using
helm uninstallto avoid accidentally deleting existing custom resources. -
Remove the CRDs.
-
List all Redpanda CRDs installed by the operator:
kubectl api-resources --api-group='cluster.redpanda.com'This command displays all CRDs defined by the Redpanda Operator. For example:
NAME SHORTNAMES APIVERSION NAMESPACED KIND redpandas rp cluster.redpanda.com/v1alpha2 true Redpanda schemas sc cluster.redpanda.com/v1alpha2 true Schema topics cluster.redpanda.com/v1alpha2 true Topic users rpu cluster.redpanda.com/v1alpha2 true User -
Delete the CRDs:
kubectl get crds -o name | grep cluster.redpanda.com | xargs kubectl deleteThis command lists all CRDs with the
cluster.redpanda.comdomain suffix and deletes them, ensuring only Redpanda CRDs are removed. Helm does not delete CRDs automatically to prevent data loss, so you must run this step manually.
-
-
(Optional) Delete any leftover PVCs or Secrets in the namespace:
The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. kubectl delete pvc,secret --all --namespace <namespace>
After completing these steps, the Redpanda Operator is no longer managing your Helm deployment.
Troubleshooting
While the deployment process can sometimes take a few minutes, a prolonged 'not ready' status may indicate an issue.
Helm v3.18.0 is not supported (json.Number error)
If you are using Helm v3.18.0, you may encounter errors such as:
Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1)
This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the Helm GitHub issue. === StatefulSet never rolls out
If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available.
To identify the Pods that are pending, use the following command:
kubectl get pod --namespace <namespace>
The response includes a list of Pods in the StatefulSet and their status.
To view logs for a specific Pod, use the following command.
kubectl logs -f <pod-name> --namespace <namespace>
You can use the output to debug your deployment.
Didn’t match pod anti-affinity rules
If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules:
Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
The Helm chart configures default podAntiAffinity rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see Number of workers.
To resolve this issue, do one of the following:
-
Create additional worker nodes.
-
Modify the anti-affinity rules (for development purposes only).
If adding nodes is not an option, you can modify the
podAntiAffinityrules in your StatefulSet to be less strict.-
Operator
-
Helm
redpanda-cluster.yamlapiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: softkubectl apply -f redpanda-cluster.yaml --namespace <namespace>-
--values
-
--set
docker-repo.yamlstatefulset: podAntiAffinity: type: softhelm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \ --values docker-repo.yamlhelm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \ --set statefulset.podAntiAffinity.type=soft -
Unable to mount volume
If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data.
-
If you’re using StorageClasses with dynamic provisioners (default), ensure they exist:
kubectl get storageclass -
If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in
storage.persistentVolume.size:kubectl get persistentvolume --namespace <namespace>
To learn how to configure different storage volumes, see Configure Storage.
Failed to pull image
When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain docker.redpanda.com is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda.
Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v<version>": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v<version>": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
To fix this error, do one of the following:
-
Replace the
image.repositoryvalue in the Helm chart withdocker.io/redpandadata/redpanda. Switching to Docker Hub avoids the rate limit issues associated withdocker.redpanda.com.-
Operator
-
Helm
redpanda-cluster.yamlapiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpandakubectl apply -f redpanda-cluster.yaml --namespace <namespace>-
--values
-
--set
docker-repo.yamlimage: repository: docker.io/redpandadata/redpandahelm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \ --values docker-repo.yamlhelm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda -
-
Authenticate to Docker Hub by logging in with your Docker Hub credentials. The
docker.redpanda.comsite acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues.
Dig not defined
This error means that you are using an unsupported version of Helm:
Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined
To fix this error, ensure that you are using the minimum required version: 3.10.0.
helm version
Repository name already exists
If you see this error, remove the redpanda chart repository, then try installing it again.
helm repo remove redpanda
helm repo add redpanda https://charts.redpanda.com
helm repo update
Fatal error during checker "Data directory is writable" execution
This error appears when Redpanda does not have write access to your configured storage volume under storage in the Helm chart.
Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test_file: permission denied
To fix this error, set statefulset.initContainers.setDataDirOwnership.enabled to true so that the initContainer can set the correct permissions on the data directories.
Cannot patch "redpanda" with kind StatefulSet
This error appears when you run helm upgrade with the --values flag but do not include all your previous overrides.
Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
To fix this error, include all the value overrides from the previous installation using either the --set or the --values flags.
Do not use the --reuse-values flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart.
|
Cannot patch "redpanda-console" with kind Deployment
This error appears if you try to upgrade your deployment and you already have console.enabled set to true.
Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
To fix this error, set console.enabled to false so that Helm doesn’t try to deploy Redpanda Console again.
Helm is in a pending-rollback state
An interrupted Helm upgrade process can leave your Helm release in a pending-rollback state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this:
-
Identify the Helm release that’s in a
pending-rollbackstate:helm list --namespace <namespace> --allLook for releases with a status of
pending-rollback. These are the ones that need intervention. -
Verify the Secret’s status to avoid affecting the wrong resource:
kubectl --namespace <namespace> get secret --show-labelsIdentify the Secret associated with your Helm release by its
pending-rollbackstatus in the labels.Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. -
Delete the Secret to clear the
pending-rollbackstate:kubectl --namespace <namespace> delete secret -l status=pending-rollback
After clearing the pending-rollback state:
-
Retry the upgrade: Restart the upgrade process. You should investigate the initial failure to avoid getting into the
pending-rollbackstate again. -
Perform a rollback: If you need to roll back to a previous release, use
helm rollback <release-name> <revision>to revert to a specific, stable release version.
Crash loop backoffs
If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it.
To prevent infinite crash loops, the Redpanda Helm chart sets the crash_loop_limit broker configuration property to 5. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the crash_loop_limit. The Pod running Redpanda remains in a CrashLoopBackoff state until its internal consecutive crash counter is reset to zero.
To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the crash_loop_sleep_sec broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue.
Example configuration:
config:
node:
crash_loop_limit: 5
crash_loop_sleep_sec: 60
In this example, when the broker hits the crash_loop_limit of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot.
To troubleshoot a crash loop backoff:
-
Check the Redpanda logs from the most recent crashes:
kubectl logs <pod-name> --namespace <namespace>Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as Loki or Datadog can capture and store logs from all containers, ensuring you have access to historical data. -
Resolve the issue that led to the crash loop backoff.
-
Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter:
-
Make changes to any of the following sections in the Redpanda Helm chart to trigger an update:
-
config.node -
config.tunable
For example:
config: node: crash_loop_limit: <new-integer> -
-
Delete the
startup_logfile in the broker’s data directory.kubectl exec <pod-name> --namespace <namespace> -- rm /var/lib/redpanda/data/startup_logIt might be challenging to execute this command within a Pod that is in a CrashLoopBackoffstate due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. -
Wait one hour since the last crash. The crash counter resets after one hour.
-
To avoid future crash loop backoffs and manage the accumulation of small segments effectively:
-
Monitor the size and number of segments regularly.
-
Optimize your Redpanda configuration for segment management.
-
Consider implementing Tiered Storage to manage data more efficiently.
A Redpanda Enterprise Edition license is required
During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade.
A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set <key>', or via Redpanda Console. To request an enterprise license, please visit <redpanda.com/upgrade>. To try Redpanda Enterprise for 30 days, visit <redpanda.com/try-enterprise>. For more information, see <https://docs.redpanda.com/current/get-started/licenses>.
If you encounter this message, follow these steps to recover:
-
Do one of the following:
-
Apply a valid Redpanda Enterprise Edition license to the cluster.
-
Disable enterprise features.
If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration.
-
-
Retry the upgrade.
For more troubleshooting steps, see Troubleshoot Redpanda in Kubernetes.
Resynchronize the bootstrap user password
|
The bootstrap user is designed to be set once at cluster creation and remain long-lived. Avoid regenerating the |
When the Redpanda Operator takes ownership of a SASL-enabled cluster, it manages a <cluster-name>-bootstrap-user Secret that holds the superuser credentials. If this Secret is regenerated after migration (for example, if you delete a Helm-era Secret and expect the operator to recreate it with clean ownership), the operator writes a new random password into the Secret and the pod environment variables, but Redpanda’s internal SCRAM database continues to hold the original password that the Helm chart set. On the next pod restart, rpk inside the pod reads the new password from its environment and fails to authenticate:
SASL_AUTHENTICATION_FAILED: Invalid credentials
The Redpanda Operator does not resynchronize this password for you: changing a SCRAM password requires authenticating with the old password, and the bootstrap Secret only tracks one credential at a time. You must update the SCRAM database manually, using the old password to authenticate and the new password from the regenerated Secret as the target.
|
If you have not regenerated the bootstrap-user Secret yet, back up the current password first so you have it available if you need to run this procedure later:
|
|
Only perform these steps if |
-
Confirm that every broker pod exposes the same
RPK_PASSvalue. The resync step in the next item updates the SCRAM database to whatever the pod’s env currently holds, so every pod must agree on that value first:for pod in $(kubectl --namespace <namespace> get pods -l app.kubernetes.io/component=redpanda-statefulset -o name); do echo "$pod: $(kubectl --namespace <namespace> exec $pod -c redpanda -- printenv RPK_PASS)" doneIf every pod reports the same password, continue to the next step.
If one or more pods report a different
RPK_PASSvalue from the rest, the bootstrap Secret on those pods' nodes is still cached by the kubelet. The Redpanda Operator creates the<cluster-name>-bootstrap-userSecret withimmutable: true, and kubelet’s local Secret cache does not always invalidate when an immutable Secret is deleted and re-created with the same name. To force affected pods to pick up the current Secret value:-
Force-delete the pod:
kubectl --namespace <namespace> delete pod <pod-name> --force --grace-period=0. -
If the env still disagrees after the pod is re-created, drain the node that hosted the pod so the StatefulSet schedules it elsewhere:
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.
Repeat the check until every pod reports the same value before continuing.
-
-
Open a shell in any broker pod. The pod’s environment already exposes the new password from the regenerated Secret:
kubectl --namespace <namespace> exec -it <cluster-name>-0 -c redpanda -- bash -
Resynchronize the SCRAM database. Replace
<old-password>with the superuser password set by the Helm chart before migration:export RPK_NEW_PASS="$RPK_PASS" (1) export RPK_PASS="<old-password>" (2) rpk acl user update "$RPK_USER" --mechanism "$RPK_SASL_MECHANISM" --new-password "$RPK_NEW_PASS" (3) export RPK_PASS="$RPK_NEW_PASS" (4) rpk cluster info (5)1 Save the new password that the operator wrote to the Secret. $RPK_PASSin the pod already points to this value.2 Switch rpkto authenticate with the original Helm-era password so the next command can reach the admin API.3 Update the superuser’s password in Redpanda’s SCRAM database to match the new password in the Secret. 4 Restore $RPK_PASSto the new password for subsequent commands.5 Verify that authentication succeeds with the new password.
After the update, the SCRAM database and the bootstrap Secret agree on the password, and subsequent pod restarts continue to authenticate cleanly.
Open an issue
If you cannot solve the issue or need assistance during the migration process, open a GitHub issue. Before opening a new issue, search the existing issues on GitHub to see if someone has already reported a similar problem or if any relevant discussions can help you.
Next steps
For information about the latest Redpanda Operator and the new Redpanda custom resource, see Redpanda in Kubernetes.