Migrate from the Redpanda Helm chart

If you are using the Redpanda Helm chart, you can migrate to the Redpanda Operator and use it to manage your Helm deployment.

The Redpanda Operator extends Kubernetes with custom resource definitions (CRDs), which allow Redpanda clusters to be treated as native Kubernetes resources. The primary resource that the Redpanda Operator uses to represent a Redpanda cluster is the Redpanda resource.

When you define a Redpanda cluster using this Redpanda resource, the Redpanda Operator takes that configuration and passes it to Flux. Flux, in turn, interacts with Helm, creating the necessary HelmRepository and HelmRelease resources to deploy and manage the Redpanda Helm chart.

Here is an example of a Redpanda custom resource:

apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: <cluster-name>
spec:
  chartRef:
    helmRepositoryName:
  clusterSpec:

  • metadata.name: Name to assign the Redpanda cluster. This name is also assigned to the Helm release.

  • spec.chartRef: Information about the Helm chart that will be used to deploy Redpanda.

  • spec.chartRef.helmRepositoryName: Name of the Helm repository where the Redpanda chart is located.

  • spec.clusterSpec: This is where you can override default values in the Redpanda Helm chart.

Prerequisites

Before migrating to the Redpanda Operator, you must have:

  • The name of your existing Helm release and the latest version of the Redpanda Helm chart that you have deployed.

    helm list -A

    In this example the chart version is 5.0.1 and the release name is redpanda.

    NAME       CHART
redpanda   redpanda-5.0.1

    Make a note of your name and version for the next steps. You’ll need to configure your Redpanda custom resource with these details.

  • Your values overrides.

    helm get values <cluster-name> --namespace <namespace>

    You should see your overrides in YAML format. You’ll need to configure your Redpanda custom resource with these details.

Before implementing any changes in your production environment, Redpanda Data recommends testing the migration in a non-production environment.

Migrate to the Redpanda Operator and Helm

To migrate to the latest Redpanda Operator and use it to manage your Helm deployment, follow these steps.

  1. Make sure that you have permission to install custom resource definitions (CRDs):

    kubectl auth can-i create CustomResourceDefinition --all-namespaces

    You should see yes in the output.

    You need these cluster-level permissions to install the Redpanda Operator CRDs in the next steps.

  2. Install the Redpanda Operator custom resource definitions (CRDs):

    kubectl kustomize "https://github.com/redpanda-data/redpanda-operator//src/go/k8s/config/crd?ref=v2.2.5-24.2.7" \
    | kubectl apply -f -

  3. Install the Redpanda Operator in the same namespace as your Redpanda Helm chart:

    helm repo add redpanda https://charts.redpanda.com
helm upgrade --install redpanda-controller redpanda/operator \
  --namespace <namespace> \
  --set image.tag=v2.2.5-24.2.7 \
  --create-namespace

  4. Ensure that the Deployment is successfully rolled out:

    kubectl --namespace <namespace> rollout status -w deployment/redpanda-controller-operator
    deployment "redpanda-controller" successfully rolled out

  5. Configure a Redpanda custom resource that Redpanda Operator will use to adopt your Redpanda cluster.

    Replace the placeholders with the values identified in the Prerequisites.

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  annotations:
    cluster.redpanda.com/managed: "true"
  creationTimestamp: null
  name: <cluster-name>
spec:
  chartRef:
    chartVersion: <chart-version>
  clusterSpec:
    <chart-overrides>
    By selecting the correct chart version, you ensure control over your deployment. If you don’t select a chart version, the operator may upgrade your chart, potentially causing unintended downtime.
    The Redpanda Operator passes these values to the Redpanda Helm chart. The keys in the clusterSpec field are the same as those in the Redpanda Helm chart. For details on the structure and configuration options of the Redpanda custom resource, refer to the Redpanda Operator CRD reference.

  6. Adopt the Redpanda cluster by creating an instance of the Redpanda custom resource in the same namespace as the Redpanda Operator:

    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

  7. Wait for the Redpanda resource to successfully reach a deployed state:

    kubectl get redpanda --namespace <namespace> --watch

    Example output:

    NAME       READY   STATUS
redpanda   True    Redpanda reconciliation succeeded

Roll back from Redpanda Operator to Helm

If you migrated to the Redpanda Operator and want to revert to using only Helm, follow these steps:

  1. Uninstall or disable the Redpanda Operator.

    You can uninstall the Redpanda Operator using Helm or disable it by changing the image to one that does not exist:

    kubectl edit pod <operator-name> --namespace <namespace>

  2. Remove the finalizers from the Redpanda, HelmRelease, HelmChart, and HelmRepository resources:

    kubectl --namespace <namespace> patch redpanda <cluster-name> -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl --namespace <namespace> patch helmrelease <cluster-name> -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl --namespace <namespace> patch helmchart <namespace>-<cluster-name> -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl --namespace <namespace> patch helmrepository <helm-repository-name> -p '{"metadata":{"finalizers":null}}' --type=merge

    Replace the <helm-repository-name> placeholder with the value of spec.chartRef.helmRepositoryName in your Redpanda resource. The default value is redpanda-repository.

  3. Delete the resources:

    kubectl delete redpanda <cluster-name> --namespace <namespace>
kubectl delete helmrelease <cluster-name> --namespace <namespace>
kubectl delete helmchart <cluster-name> --namespace <namespace>
kubectl delete helmrepository <cluster-name> --namespace <namespace>

After completing these steps, the Redpanda Operator is no longer managing your Helm deployment.

Troubleshooting

While the deployment process can sometimes take a few minutes, a prolonged 'not ready' status may indicate an issue.

HelmRelease is not ready

If you are using the Redpanda Operator, you may see the following message while waiting for a Redpanda custom resource to be deployed:

NAME       READY   STATUS
redpanda   False   HelmRepository 'redpanda/redpanda-repository' is not ready
redpanda   False   HelmRelease 'redpanda/redpanda' is not ready

While the deployment process can sometimes take a few minutes, a prolonged 'not ready' status may indicate an issue. Follow the steps below to investigate:

  1. Check the status of the HelmRelease:

    kubectl describe helmrelease <redpanda-resource-name> --namespace <namespace>

  2. Review the Redpanda Operator logs:

    kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace <namespace>

HelmRelease retries exhausted

The HelmRelease retries exhausted error occurs when the Helm Controller has tried to reconcile the HelmRelease a number of times, but these attempts have failed consistently.

The Helm Controller watches for changes in HelmRelease objects. When changes are detected, it tries to reconcile the state defined in the HelmRelease with the state in the cluster. The process of reconciliation includes installation, upgrade, testing, rollback or uninstallation of Helm releases.

You may see this error due to:

  • Incorrect configuration in the HelmRelease.

  • Issues with the chart, such as a non-existent chart version or the chart repository not being accessible.

  • Missing dependencies or prerequisites required by the chart.

  • Issues with the underlying Kubernetes cluster, such as insufficient resources or connectivity issues.

To debug this error do the following:

  1. Check the status of the HelmRelease:

    kubectl describe helmrelease <cluster-name> --namespace <namespace>

  2. Review the Redpanda Operator logs:

    kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace <namespace>

When you find and fix the error, you must use the Flux CLI, fluxctl, to suspend and resume the reconciliation process:

  1. Install Flux CLI.

  2. Suspend the HelmRelease:

    flux suspend helmrelease <cluster-name> --namespace <namespace>

  3. Resume the HelmRelease:

    flux resume helmrelease <cluster-name> --namespace <namespace>

StatefulSet never rolls out

If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available.

To identify the Pods that are pending, use the following command:

kubectl get pod --namespace <namespace>

The response includes a list of Pods in the StatefulSet and their status.

To view logs for a specific Pod, use the following command.

kubectl logs -f <pod-name> --namespace <namespace>

You can use the output to debug your deployment.

Unable to mount volume

If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data.

  • If you’re using StorageClasses with dynamic provisioners (default), ensure they exist:

    kubectl get storageclass

  • If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in storage.persistentVolume.size:

    kubectl get persistentvolume --namespace <namespace>

To learn how to configure different storage volumes, see Configure Storage.

Dig not defined

This error means that you are using an unsupported version of Helm:

Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined

Ensure that you are using the minimum required version: 3.6.0.

helm version

Repository name already exists

If you see this error, remove the redpanda chart repository, then try installing it again.

helm repo remove redpanda
helm repo add redpanda https://charts.redpanda.com
helm repo update

Fatal error during checker "Data directory is writable" execution

This error appears when Redpanda does not have write access to your configured storage volume under storage in the Helm chart.

Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test_file: permission denied

To fix this error, set statefulset.initContainers.setDataDirOwnership.enabled to true so that the initContainer can set the correct permissions on the data directories.

Cannot patch "redpanda" with kind StatefulSet

This error appears when you run helm upgrade with the --values flag but do not include all your previous overrides.

Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Ensure to do one of the following:

  • Include all the value overrides from the previous installation or upgrade using either the --set or the --values flags.

  • Use the --reuse-values flag.

    Do not use the --reuse-values flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart.

Cannot patch "redpanda-console" with kind Deployment

This error appears if you try to upgrade your deployment and you already have console.enabled set to true.

Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

To fix this error, set console.enabled to false so that Helm doesn’t try to deploy Redpanda Console again.

For more troubleshooting steps, see Troubleshoot Redpanda in Kubernetes.

Open an issue

If you cannot solve the issue or need assistance during the migration process, open a GitHub issue. Before opening a new issue, search the existing issues on GitHub to see if someone has already reported a similar problem or if any relevant discussions can help you.

Next steps

For information about the latest Redpanda Operator and the new Redpanda custom resource, see Redpanda in Kubernetes.