# Upgrade Redpanda in Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Upgrade Redpanda in Kubernetes latest-redpanda-tag: v26.1.9 latest-console-tag: v3.7.3 latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: March 31, 2027 latest-connect-version: 4.93.0 docname: k-rolling-upgrade page-component-name: streaming page-version: "26.1" page-component-version: "26.1" page-component-title: Streaming page-relative-src-path: k-rolling-upgrade.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/upgrade/pages/k-rolling-upgrade.adoc description: To benefit from Redpanda's new features and enhancements, upgrade to the latest version. page-git-created-date: "2023-08-04" page-git-modified-date: "2026-01-28" support-status: supported --- To benefit from Redpanda’s new features and enhancements, upgrade to the latest version. New features are available after all brokers in the cluster are upgraded and restarted. Redpanda Streaming version numbers follow the convention AB.C.D, where AB is the two digit year, C is the feature release, and D is the patch release. For example, v22.3.1 indicates the first patch release on the third feature release of the year 2022. Patch releases include bug fixes and minor improvements, with no change to user-facing behavior. New and enhanced features are documented with each feature release. You can find a list of all releases on [GitHub](https://github.com/redpanda-data/redpanda/releases). ## [](#limitations)Limitations The following limitations ensure a smooth transition between versions and help maintain your cluster’s stability. - **Broker upgrades**: - New features are enabled only after upgrading all brokers in the cluster. - You can upgrade only one feature release at a time, for example from 22.2 to 22.3. Skipping feature releases is not supported. - If you have [enterprise features](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/) enabled without a valid license, Redpanda fails to upgrade to new feature releases, starting from version 24.3. - **Rollbacks**: You can roll back to the original version only if at least one broker is still running the original version (not yet upgraded) and the cluster hasn’t yet restarted. - **Downgrades**: Downgrades are possible only between patch releases of the same feature release. For example, you can downgrade from 22.2.2 to 22.2.1. Downgrading to previous feature releases, such as 22.1.x, is not supported. - **Tiered Storage**: If you have [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) enabled and you’re upgrading to 23.2, object storage uploads are paused until all brokers are upgraded. If the cluster cannot upgrade, roll it back to the original version. > ⚠️ **CAUTION** > > In a mixed-version state, the cluster could run out of disk space. If you need to force a mixed-version cluster to upload, transfer partition leadership to brokers running the original version. - **Remote Read Replicas**: Upgrade the Remote Read Replica cluster first, ensuring it’s on the same version as the origin cluster or one feature release ahead of the origin cluster. When upgrading to Redpanda 23.2, metadata from object storage is not synchronized until all brokers in the cluster are upgraded. If you need to force a mixed-version cluster to sync read replicas, transfer partition leadership to brokers running the original version. ## [](#prerequisites)Prerequisites - [A Redpanda cluster running in Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/). - The default [RollingUpdate strategy](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/#statefulset-updatestrategy-type) configured in the Helm values. ### [](#review-incompatible-changes)Review incompatible changes - **Breaking changes in Redpanda 26.1**: - If FIPS mode is enabled, change any SASL/SCRAM user passwords shorter than 14 characters to at least 14 characters before upgrading. FIPS 140-3 enforces stricter HMAC key size requirements than FIPS 140-2. Because Redpanda stores passwords in encrypted form, it cannot check the length of existing passwords. Clients with passwords shorter than 14 characters will fail to authenticate after the upgrade. See [Configure Redpanda for FIPS](https://docs.redpanda.com/streaming/current/manage/security/fips-compliance/). - **Breaking changes in Redpanda 25.3**: - Redpanda v25.3 introduces breaking schema changes for Iceberg topics. If you are using Iceberg topics and want to retain the data in the corresponding Iceberg tables, review [Schema Changes and Migration Guide for Iceberg Topics in Redpanda v25.3](https://docs.redpanda.com/streaming/current/upgrade/iceberg-schema-changes-and-migration-guide/) before upgrading your cluster, and follow the required migration steps to avoid sending new records to a dead-letter queue table. - Schema Registry no longer allows specifying a schema ID and version when registering a schema in read-write mode. You must use import mode to register a schema with a specific ID and version. See [Use the Schema Registry API](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-api/#set-schema-registry-mode) for more information. - Starting in Redpanda v25.3, several topic properties support enhanced tristate behavior. Properties like `retention.ms`, `retention.bytes`, `segment.ms`, and others now distinguish between zero values (immediate eligibility for cleanup/compaction) and negative values (disable the feature entirely). Previously, zero and negative values were treated the same way. For the complete list of affected properties and detailed information, see [Redpanda v26.1 behavior changes](https://docs.redpanda.com/streaming/current/get-started/release-notes/redpanda/#behavior-changes). Review your topic configurations if you currently use zero values for these properties. - **Breaking change in Redpanda 25.2:** Ephemeral credentials for HTTP Proxy are removed. If your HTTP Proxy API listeners use [`authentication_method`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#http_proxy_auth_method): `none`, you must configure explicit SASL credentials ([`scram_username`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#scram_username), [`scram_password`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#scram_password), and [`sasl_mechanism`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#sasl_mechanism)) for HTTP Proxy to authenticate with the Kafka API. > ⚠️ **WARNING** > > This allows any HTTP API user to access Kafka using shared credentials. Redpanda Data recommends enabling [HTTP Proxy authentication](https://docs.redpanda.com/streaming/current/develop/http-proxy/#authenticate-with-http-proxy) instead. For configuration instructions, see [Configure HTTP Proxy to connect to Redpanda with SASL](https://docs.redpanda.com/streaming/current/manage/security/authentication/#schema-and-http-to-redpanda). For details about this breaking change, see [What’s new](https://docs.redpanda.com/streaming/current/get-started/release-notes/redpanda/#http-proxy-authentication-changes). - Redpanda Console v3.0.0 introduces breaking changes. If you are using Redpanda Console v2.x, [review the migration guide](https://docs.redpanda.com/streaming/current/migrate/console-v3/) to address breaking changes before upgrading Redpanda Console. - Starting in version 24.2, when managing configuration properties using the AlterConfigs API directly, Redpanda resets all unspecified values to the default values. This aligns more closely with the behavior in Apache Kafka. There is no change if you’re managing your configuration with tools like `rpk`, Redpanda Console, Kubernetes, Helm, or Terraform. > 📝 **NOTE** > > This does not pertain to the `redpanda.remote.` topic properties, such as `redpanda.remote.delete`. The remote properties are not reset to their defaults by the AlterConfigs API to avoid the possibility of unintentionally disabling Tiered Storage for a topic, which could cause significant operational issues for clusters designed to use Tiered Storage exclusively or with object storage as the only durable storage tier. - Starting in version 24.2, [client throughput limits](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/#client-throughput-limits) are compatible with the [AlterClientQuotas and DescribeClientQuotas](https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+Client+Quota+APIs+to+the+Admin+Client) Kafka APIs. Redpanda determines client throughput limits on a per-broker basis. In earlier versions, client throughput quotas were applied from cluster configurations on a per-shard basis. - Starting in version 24.2, [transaction\_max\_timeout\_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#transaction_max_timeout_ms) defaults to 15 minutes, which limits the timeout a transactional producer can set during initialization. When performing upgrades to version 24.2, if you have clients with timeouts larger than `transaction_max_timeout_ms`, then new producer initialization requests will fail due to this limit enforcement. Either reduce the client set transaction timeout to less than 15 minutes or increase the default `transaction_max_timeout_ms` on the service side. - Patch releases in [22.3.14](https://github.com/redpanda-data/redpanda/discussions/9522) and [23.1.2](https://github.com/redpanda-data/redpanda/discussions/9523) changed the behavior when remote read is disabled and the requested Raft term falls below the local log’s beginning. In earlier versions, Redpanda returned an offset -1. With the patch, when you request a value older than the lowest offset, Redpanda returns the lowest offset, not -1. ## [](#impact-of-broker-restarts)Impact of broker restarts When brokers restart, clients may experience higher latency, nodes may experience CPU spikes when the broker becomes available again, and you may receive alerts about under-replicated partitions. Topics that weren’t using replication (that is, topics that had `replication.factor=1`) will be unavailable. ### [](#temporary-increase-in-latency-on-clients-producers-and-consumers)Temporary increase in latency on clients (producers and consumers) When you restart one or more brokers in a cluster, clients (consumers and producers) may experience higher latency due to partition leadership reassignment. Because clients must communicate with the leader of a partition, they may send a request to a broker whose leadership has been transferred, and receive `NOT_LEADER_FOR_PARTITION`. In this case, clients must request metadata from the cluster to find out the address of the new leader. Clients refresh their metadata periodically, or when the client receives some retryable errors that indicate that the metadata may be stale. For example: 1. Broker A shuts down. 2. Client sends a request to broker A, and receives `NOT_LEADER_FOR_PARTITION`. 3. Client requests metadata, and learns that the new leader is broker B. 4. Client sends the request to broker B. ### [](#cpu-spikes-upon-broker-restart)CPU spikes upon broker restart When a restarted broker becomes available again, you may see your nodes' CPU usage increase temporarily. This temporary increase in CPU usage is due to the cluster rebalancing the partition replicas. ### [](#under-replicated-partitions)Under-replicated partitions When a broker is in maintenance mode, Redpanda continues to replicate updates to that broker. When a broker is taken offline during a restart, partitions with replicas on the broker could become out of sync until it is brought back online. Once the broker is available again, data is copied to its under-replicated replicas until all affected partitions are in sync with the partition leader. ## [](#check-your-current-redpanda-version)Check your current Redpanda version Before you perform a rolling upgrade: - Find the Redpanda version that you are currently running. To find your current version of Redpanda, use [`rpk redpanda admin brokers list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-admin-brokers-list/): ```bash kubectl exec --namespace -c redpanda -- \ rpk redpanda admin brokers list ``` Expected output: The Redpanda version for each broker is listed under `VERSION`. ```none ID VERSION 0 v26.1.9 1 v26.1.9 2 v26.1.9 ``` - [Review the Kubernetes compatibility matrix](https://docs.redpanda.com/streaming/current/upgrade/k-compatibility/) to find out if you need to upgrade the Helm chart or the Redpanda Operator to use your chosen version of Redpanda. If your current version of Redpanda is more than one feature release behind the one to which you want to upgrade, you must first upgrade to an intermediate version of Redpanda. - Learn what’s changed since your original version. To find information about what has changed between Redpanda versions, check the [release notes](https://github.com/redpanda-data/redpanda/releases). ## [](#check-license-status)Check license status If you use [enterprise features](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/#self-managed) of Redpanda, make sure that you have a valid Enterprise Edition license key. From version 24.3, Redpanda fails to upgrade to a new feature release if you have enterprise features enabled without a valid license key. ```bash kubectl exec --namespace -c redpanda -- \ rpk cluster license info ``` This command will display the current license details, including the expiration date and whether any enterprise features are active. For example: LICENSE INFORMATION =================== Organization: redpanda Type: enterprise Expires: Oct 24 2027 > 📝 **NOTE** > > Redpanda blocks upgrades to new feature releases if enterprise features are active without a valid license. Ensure compliance by obtaining a license to maintain access to the latest features and updates. ## [](#prepare-your-cluster)Prepare your cluster Before you upgrade, you must make sure that your cluster is in a healthy state and that your topics are configured to limit outages during the upgrade process. 1. Check for topics that have a replication factor greater than one. If you have topics with a replication factor of 1, and if you have sufficient disk space, temporarily [increase the replication factor](https://docs.redpanda.com/streaming/current/migrate/data-migration/#change-topic-replication-factor) to limit outages for these topics during the rolling upgrade. Increase the replication factor before you upgrade to ensure that Redpanda has time to replicate data to other brokers. 2. Ensure that the cluster is healthy: ```bash kubectl exec --namespace -c redpanda -- \ rpk cluster health ``` The draining process won’t start until the cluster is healthy. Example output: CLUSTER HEALTH OVERVIEW ======================= Healthy: true **(1)** Controller ID: 0 All nodes: \[0 1 2\] **(2)** Nodes down: \[\] **(3)** Leaderless partitions: \[\] **(3)** Under-replicated partitions: \[\] **(3)** | 1 | The cluster is either healthy (true) or unhealthy (false). | | --- | --- | | 2 | The node IDs of all brokers in the cluster. | | 3 | These fields contain data only when the cluster is unhealthy. | ## [](#upgrade)Perform a rolling upgrade Performing a rolling upgrade allows you to update the version of Redpanda without downtime. This process ensures that each broker is sequentially updated and restarted, minimizing the impact on your environment. The upgrade process depends on your deployment method: - **Redpanda Operator**: Update the Redpanda custom resource to specify a new Helm chart version or Redpanda image tag. The operator manages the upgrade process. - **Helm**: Either upgrade to a newer version of the Redpanda Helm chart (recommended) or update the Redpanda image tag in your existing Helm release. Upgrading the chart is preferred because it ensures that any new configuration parameters are included. Upgrading a Redpanda cluster in Kubernetes triggers a sequential restart of the Pods managed by the StatefulSet. During each broker’s restart, the following steps occur: - The `preStop` lifecycle hook is executed to place the broker into maintenance mode. This step ensures that the broker stops accepting new connections and finishes processing its current tasks. - Kubernetes then terminates the Pod. If the broker does not shut down within the allowed grace period (default 90 seconds), Kubernetes forcefully terminates it using a `SIGKILL` signal. - After the Pod is terminated, the `postStart` lifecycle hook is executed to take the broker out of maintenance mode, allowing it to rejoin the cluster once restarted. ### Operator 1. [Review the Kubernetes compatibility matrix](https://docs.redpanda.com/streaming/current/upgrade/k-compatibility/) and [review the Redpanda Operator release notes](https://github.com/redpanda-data/redpanda-operator/releases) to determine the version of the Redpanda Operator that is compatible with the Helm chart and Redpanda version you plan to use. The Redpanda Operator and Helm chart follow the same versioning scheme as Redpanda core (for example, operator version 25.1.1 deploys Redpanda 25.1 by default). Each operator version supports the corresponding Redpanda version ±1 minor version. 2. If you need to upgrade the Redpanda Operator: 1. Back up your current Helm values for the Redpanda Operator: ```bash helm get values redpanda-controller --namespace > redpanda-operator-values-backup.yaml ``` 2. Upgrade the Redpanda Operator to a version compatible with your target Redpanda version: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade redpanda-controller redpanda/operator \ --namespace \ --version \ --set crds.enabled=true ``` Replace `` with the version of the Redpanda Operator you want to upgrade to. 3. Verify that the operator deployment is successfully rolled out: ```bash kubectl --namespace rollout status --watch deployment/redpanda-controller-operator ``` Expected output deployment "redpanda-controller-operator" successfully rolled out 3. Upgrade the Redpanda cluster by updating the Redpanda image tag in your Redpanda resource: `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: image: tag: (1) statefulset: # Optional podTemplate: spec: terminationGracePeriodSeconds: (2) ``` | 1 | The version of Redpanda to deploy (for example, v25.2.1). The operator automatically uses the appropriate Helm chart version for the specified Redpanda version. | | --- | --- | | 2 | The statefulset.podTemplate.spec.terminationGracePeriodSeconds setting defines how long Kubernetes will wait for the broker to shut down gracefully before forcefully terminating it. The default value is 90 seconds, which is enough for most clusters, but might require adjustment based on your workload. Modify this setting if your Redpanda brokers have high loads or hold large amounts of data, as they might need more time to shut down gracefully. | 4. Apply the Redpanda resource to deploy the Redpanda cluster: ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` ### Helm 1. [Review the Kubernetes compatibility matrix](https://docs.redpanda.com/streaming/current/upgrade/k-compatibility/) and verify which version of the Redpanda Helm chart supports the Redpanda version you plan to upgrade to. The Helm chart version can dictate which configurations and Kubernetes resources are available or required for that specific version of Redpanda. 2. Check the default Redpanda version of a chart to make sure that it uses the version that you want to upgrade your cluster to. ```bash helm show chart --version redpanda/redpanda | grep "appVersion" ``` Replace `` with the version number of a newer chart. 3. Back up your current Helm values for the Redpanda Helm chart: ```bash helm get values redpanda --namespace > redpanda-values-backup.yaml ``` You’ll need to apply these overrides in the next step. 4. Optional: Update the following settings: `redpanda-version.yaml` ```yaml image: tag: (1) statefulset: podTemplate: spec: terminationGracePeriodSeconds: (2) ``` | 1 | If you need to upgrade to an intermediate version of Redpanda, use this setting to specify the version of Redpanda to deploy. This version overrides the default one in the Helm chart. Replace with a valid version tag. | | --- | --- | | 2 | The statefulset.podTemplate.spec.terminationGracePeriodSeconds setting defines how long Kubernetes will wait for the broker to shut down gracefully before forcefully terminating it. The default value is 90 seconds, which is enough for most clusters, but might require adjustment based on your workload. Modify this setting in your Helm values file if your Redpanda brokers have high loads or hold large amounts of data, as they might need more time to shut down gracefully. | 5. If your current chart uses Redpanda Console v2, migrate your configuration to the new v3 format before upgrading to 25.1.x. For more information, see [Migrate to Redpanda Console v3.0.0](https://docs.redpanda.com/streaming/current/migrate/console-v3/). 6. Deploy Redpanda with the new Helm chart version: ```bash helm upgrade --install redpanda redpanda/redpanda --namespace \ --create-namespace \ --version \ --values redpanda-version.yaml ``` Make sure to include all existing overrides, otherwise the upgrade may fail. For example, if you already enabled SASL, include the same SASL overrides. > ⚠️ **CAUTION** > > Do not use the `--reuse-values` flag, otherwise Helm won’t include any new values from the upgraded chart. ## [](#verify-the-upgrade)Verify the upgrade After upgrading, verify that your Redpanda cluster is functioning correctly: 1. Wait for the Pods to be terminated and recreated with the new version of Redpanda. ```bash kubectl get pod --namespace --watch ``` Each Pod in the StatefulSet is terminated one at a time, starting from the one with the highest ordinal. Example output NAME READY STATUS redpanda-controller-operator 2/2 Running redpanda-0 2/2 Running redpanda-1 2/2 Running redpanda-2 0/2 Init:0/3 redpanda-configuration-88npt 0/1 Completed redpanda-console-7cf85cf87f-rmtnj 1/1 Running redpanda-post-upgrade-ljqpr 0/1 Completed 2. When all of the Pods are ready and have a `Running` status, verify that the brokers are now running the upgraded version of Redpanda: ```bash kubectl exec --namespace -c redpanda -- \ rpk redpanda admin brokers list ``` 3. If you upgraded from operator v25.1.x to v25.2.x or later, check for Console migration warnings: When upgrading across this version boundary, Redpanda Console is automatically upgraded from v2 to v3. The operator attempts to migrate your Console configuration automatically. If any configuration settings cannot be migrated, warnings are added to the Console custom resource. ```bash kubectl get console --namespace -o jsonpath='{.spec.warnings}' ``` If warnings are present, review them and manually update your Console configuration as needed. For details about migrating to Console v3, see [Migrate to Redpanda Console v3.0.0](https://docs.redpanda.com/streaming/current/migrate/console-v3/). ## [](#roll-back)Roll back If something does not go as planned during a rolling upgrade, you can roll back to the original version as long as you have not upgraded all brokers. The StatefulSet uses the `RollingUpdate` strategy by default in [`statefulset.updateStrategy.type`](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/#statefulsetupdatestrategytype), which means all Pods in the StatefulSet are restarted in reverse-ordinal order. For details, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies). ### Operator Revert the change that you made to the version and reapply the manifest. ### Helm 1. Find the previous revision: ```bash helm history redpanda --namespace ``` Example output REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Fri Mar 3 15:16:24 year superseded redpanda-2.12.2 v22.3.13 Install complete 2 Fri Mar 3 15:19:41 year deployed redpanda-2.12.2 v22.3.13 Upgrade complete 2. Roll back to the previous revision: ```bash helm rollback redpanda --namespace ``` 3. Verify that the cluster is healthy. If the cluster is unhealthy, the upgrade may still be in progress. The command exits when the cluster is healthy. ```bash kubectl exec --namespace -c redpanda -- \ rpk cluster health \ --watch --exit-when-healthy ``` Example output: ```none CLUSTER HEALTH OVERVIEW ======================= Healthy: true Controller ID: 1 All nodes: [2,1,0] Nodes down: [] Leaderless partitions: [] ``` ## [](#troubleshooting)Troubleshooting ### [](#helm-v3-18-0-is-not-supported-json-number-error)Helm v3.18.0 is not supported (json.Number error) If you are using Helm v3.18.0, you may encounter errors such as: Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). === StatefulSet never rolls out If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available. To identify the Pods that are pending, use the following command: ```bash kubectl get pod --namespace ``` The response includes a list of Pods in the StatefulSet and their status. To view logs for a specific Pod, use the following command. ```bash kubectl logs -f --namespace ``` You can use the output to debug your deployment. ### [](#didnt-match-pod-anti-affinity-rules)Didn’t match pod anti-affinity rules If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules: Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). To resolve this issue, do one of the following: - Create additional worker nodes. - Modify the anti-affinity rules (for development purposes only). If adding nodes is not an option, you can modify the `podAntiAffinity` rules in your StatefulSet to be less strict. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: soft ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml statefulset: podAntiAffinity: type: soft ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set statefulset.podAntiAffinity.type=soft ``` ### [](#unable-to-mount-volume)Unable to mount volume If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data. - If you’re using StorageClasses with dynamic provisioners (default), ensure they exist: ```bash kubectl get storageclass ``` - If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in `storage.persistentVolume.size`: ```bash kubectl get persistentvolume --namespace ``` To learn how to configure different storage volumes, see [Configure Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/). ### [](#failed-to-pull-image)Failed to pull image When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain `docker.redpanda.com` is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda. Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit To fix this error, do one of the following: - Replace the `image.repository` value in the Helm chart with `docker.io/redpandadata/redpanda`. Switching to Docker Hub avoids the rate limit issues associated with `docker.redpanda.com`. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpanda ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml image: repository: docker.io/redpandadata/redpanda ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda ``` - Authenticate to Docker Hub by logging in with your Docker Hub credentials. The `docker.redpanda.com` site acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues. ### [](#dig-not-defined)Dig not defined This error means that you are using an unsupported version of [Helm](https://helm.sh/docs/intro/install/): Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined To fix this error, ensure that you are using the minimum required version: 3.10.0. ```bash helm version ``` ### [](#repository-name-already-exists)Repository name already exists If you see this error, remove the `redpanda` chart repository, then try installing it again. ```bash helm repo remove redpanda helm repo add redpanda https://charts.redpanda.com helm repo update ``` ### [](#fatal-error-during-checker-data-directory-is-writable-execution)Fatal error during checker "Data directory is writable" execution This error appears when Redpanda does not have write access to your configured storage volume under `storage` in the Helm chart. Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test\_file: permission denied To fix this error, set `statefulset.initContainers.setDataDirOwnership.enabled` to `true` so that the initContainer can set the correct permissions on the data directories. ### [](#cannot-patch-redpanda-with-kind-statefulset)Cannot patch "redpanda" with kind StatefulSet This error appears when you run `helm upgrade` with the `--values` flag but do not include all your previous overrides. Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden To fix this error, include all the value overrides from the previous installation using either the `--set` or the `--values` flags. > ⚠️ **WARNING** > > Do not use the `--reuse-values` flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart. ### [](#cannot-patch-redpanda-console-with-kind-deployment)Cannot patch "redpanda-console" with kind Deployment This error appears if you try to upgrade your deployment and you already have `console.enabled` set to `true`. Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map\[string\]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:\[\]v1.LabelSelectorRequirement(nil)}: field is immutable To fix this error, set `console.enabled` to `false` so that Helm doesn’t try to deploy Redpanda Console again. ### [](#helm-is-in-a-pending-rollback-state)Helm is in a pending-rollback state An interrupted Helm upgrade process can leave your Helm release in a `pending-rollback` state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this: 1. Identify the Helm release that’s in a `pending-rollback` state: ```bash helm list --namespace --all ``` Look for releases with a status of `pending-rollback`. These are the ones that need intervention. 2. Verify the Secret’s status to avoid affecting the wrong resource: ```bash kubectl --namespace get secret --show-labels ``` Identify the Secret associated with your Helm release by its `pending-rollback` status in the labels. > ⚠️ **WARNING** > > Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. 3. Delete the Secret to clear the `pending-rollback` state: ```bash kubectl --namespace delete secret -l status=pending-rollback ``` After clearing the `pending-rollback` state: - **Retry the upgrade**: Restart the upgrade process. You should investigate the initial failure to avoid getting into the `pending-rollback` state again. - **Perform a rollback**: If you need to roll back to a previous release, use `helm rollback ` to revert to a specific, stable release version. ### [](#crash-loop-backoffs)Crash loop backoffs If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it. To prevent infinite crash loops, the Redpanda Helm chart sets the [`crash_loop_limit`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_limit) broker configuration property to `5`. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the `crash_loop_limit`. The Pod running Redpanda remains in a `CrashLoopBackoff` state until its internal consecutive crash counter is reset to zero. To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the [`crash_loop_sleep_sec`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_sleep_sec) broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue. Example configuration: ```yaml config: node: crash_loop_limit: 5 crash_loop_sleep_sec: 60 ``` In this example, when the broker hits the `crash_loop_limit` of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot. To troubleshoot a crash loop backoff: 1. Check the Redpanda logs from the most recent crashes: ```bash kubectl logs --namespace ``` > 📝 **NOTE** > > Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as [Loki](https://grafana.com/docs/loki/latest/) or [Datadog](https://www.datadoghq.com/product/log-management/) can capture and store logs from all containers, ensuring you have access to historical data. 2. Resolve the issue that led to the crash loop backoff. 3. Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter: - Make changes to any of the following sections in the Redpanda Helm chart to trigger an update: - `config.node` - `config.tunable` For example: ```yaml config: node: crash_loop_limit: ``` - Delete the `startup_log` file in the broker’s data directory. ```bash kubectl exec --namespace -- rm /var/lib/redpanda/data/startup_log ``` > 📝 **NOTE** > > It might be challenging to execute this command within a Pod that is in a `CrashLoopBackoff` state due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. - Wait one hour since the last crash. The crash counter resets after one hour. To avoid future crash loop backoffs and manage the accumulation of small segments effectively: - [Monitor](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) the size and number of segments regularly. - Optimize your Redpanda configuration for segment management. - Consider implementing [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) to manage data more efficiently. ### [](#a-redpanda-enterprise-edition-license-is-required)A Redpanda Enterprise Edition license is required During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade. A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set ', or via Redpanda Console. To request an enterprise license, please visit . To try Redpanda Enterprise for 30 days, visit . For more information, see . If you encounter this message, follow these steps to recover: 1. [Roll back the affected broker to the original version](#roll-back). 2. Do one of the following: - [Apply a valid Redpanda Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) to the cluster. - Disable enterprise features. If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration. 3. Retry the upgrade. ## [](#suggested-reading)Suggested reading Set up a real-time dashboard to monitor your cluster health, see [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/). ## Suggested labs - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) [Search all labs](https://docs.redpanda.com/labs)