Upgrade Redpanda

Loading...

To benefit from Redpanda's new features and enhancements, upgrade to the latest version. Redpanda recommends that you perform a rolling upgrade on production clusters, which requires all brokers to be placed into maintenance mode and restarted separately, one after the other.

Redpanda platform version numbers follow the convention AB.C.D, where AB is the two digit year, C is the feature release, and D is the patch release. For example, version 22.3.1 indicates the first patch release on the third feature release of the year 2022. Patch releases include bug fixes and minor improvements, with no change to user-facing behavior. New and enhanced features are documented with each feature release.

important

• New features are enabled after all brokers (nodes) in the cluster are upgraded. You can stop the upgrade process and roll back to the original version as long as you have not upgraded every broker and restarted the cluster.
• Redpanda only supports upgrading one sequential feature release at a time. For example, you can upgrade from the 22.2 feature release to 22.3. You cannot skip feature releases.
• Redpanda only supports downgrading between sequential patch releases. For example, you can downgrade from the 22.2.2 patch release to 22.2.1, but you cannot downgrade to 22.1.7.
• With Remote Read Replicas, upgrade the Remote Read Replica cluster before upgrading the origin cluster. The Remote Read Replica cluster must run on the same version of Redpanda as the origin cluster, or just one feature release ahead of the origin cluster.

Prerequisites

• A running Redpanda cluster.
• jq for listing available versions.
• An understanding of the impact of broker restarts on clients, node CPU, and any alerting systems you use.

Find a new version

Before you upgrade, find out which Redpanda version you are currently running, whether you can upgrade straight to the new version, and what's changed since your original version. To find your current version, run:

Linux

rpk redpanda admin brokers list

For all available flags, see the rpk redpanda admin brokers list command reference.

Docker

caution

Running Redpanda directly on Docker is not supported for production usage. This platform should only be used for testing.

docker exec -it <container_name><container_tag> rpk version 

Remember to replace the variables <container_name> and <container_tag>. The container tag determines which version of rpk to use. The release process bundles rpk and Redpanda into the same container tag with the same version.

macOS

brew list --versions | grep redpanda

Example output

v22.3.11 (rev 9eefb90)

note

If your current version is more than one feature release behind the latest Redpanda version, you must first upgrade to an intermediate version. To list all available versions:

curl -s 'https://hub.docker.com/v2/repositories/redpandadata/redpanda/tags/?ordering=last_updated&page=1&page_size=50' | jq -r '.results[].name'

Check the release notes to find information about what has changed between Redpanda versions.

Impact of broker restarts

When brokers restart, clients may experience higher latency, nodes may experience CPU spikes when the broker becomes available again, and you may receive alerts about under-replicated partitions. Topics that weren't using replication (that is, topics that had replication.factor=1) will be unavailable.

Temporary increase in latency on clients (producers and consumers)

When you restart one or more brokers in a cluster, clients (consumers and producers) may experience higher latency due to partition leadership reassignment. Because clients must communicate with the leader of a partition, they may send a request to a broker whose leadership has been transferred, and receive NOT_LEADER_FOR_PARTITION. In this case, clients must request metadata from the cluster to find out the address of the new leader. Clients refresh their metadata periodically, or when the client receives some retryable errors that indicate that the metadata may be stale. For example:

1. Broker A shuts down.
2. Client sends a request to broker A, and receives NOT_LEADER_FOR_PARTITION.
3. Client requests metadata, and learns that the new leader is broker B.
4. Client sends the request to broker B.

CPU spikes upon broker restart

When a restarted broker becomes available again, you may see your nodes' CPU usage increase temporarily. This temporary increase in CPU usage is due to the cluster rebalancing the partition replicas.

Under-replicated partitions

When a broker is in maintenance mode, Redpanda continues to replicate updates to that broker. When a broker is taken offline during a restart, partitions with replicas on the broker could become out of sync until it is brought back online. Once the broker is available again, data is copied to its under-replicated replicas until all affected partitions are in sync with the partition leader.

Perform a rolling upgrade

A rolling upgrade involves putting a broker into maintenance mode, upgrading the broker, taking the broker out of maintenance mode, and then repeating the process on the next broker in the cluster. Placing brokers into maintenance mode ensures a smooth upgrade of your cluster while reducing the risk of interruption or degradation in service.

When a broker is placed into maintenance mode, it reassigns its partition leadership to other brokers for all topics that have a replication factor greater than one. Reassigning partition leadership involves draining leadership from the broker and transferring that leadership to another broker. If you have topics with replication.factor=1, and if you have sufficient disk space, Redpanda recommends temporarily increasing the replication factor. This can help limit outages for these topics during the rolling upgrade. Do this before the upgrade to make sure there's time for the data to replicate to other brokers. For more information, see Change topic replication factor.

To ensure that all brokers are active before upgrading, run:

rpk redpanda admin brokers list

All brokers should show active for MEMBERSHIP-STATUS and true for IS-ALIVE:

Example output

NODE-ID  NUM-CORES  MEMBERSHIP-STATUS  IS-ALIVE  BROKER-VERSION
0        1          active             true      v22.3.11
1        1          active             true      v22.3.11
2        1          active             true      v22.3.11

New features in a version are enabled after all brokers in the cluster are upgraded. If problems occur, the upgrade is not committed.

note

Redpanda started supporting consumer offsets in version 22.1. When upgrading from version 21.11 to 22.1, after all brokers are upgraded, Redpanda starts to migrate consumer group topics to __consumer_offsets. This takes some time, depending on the data size. Until it finishes, all consumer group-related operations (consume, offset commit, coordinator election) are blocked. The migration to consumer offsets is complete when you see consumer offset feature enabled in all brokers.

Enable maintenance mode

1. Check that all brokers are healthy:

   rpk cluster health
   Expected output

   CLUSTER HEALTH OVERVIEW
   =======================
   Healthy:                     true
   Controller ID:               0
   All nodes:                   [0 1 2]
   Nodes down:                  []
   Leaderless partitions:       []
   Under-replicated partitions: []

2. Select a broker that has not been upgraded yet and place it into maintenance mode:

   rpk cluster maintenance enable <node-id> --wait

   The --wait option tells the command to wait until a given broker, 0 in this example, finishes draining all partitions it originally served. After the partition draining completes, the command completes.

   Expected output

   Successfully enabled maintenance mode for node 0
   Waiting for node to drain...
   NODE-ID  DRAINING  FINISHED  ERRORS  PARTITIONS  ELIGIBLE  TRANSFERRING  FAILED
   0        false     false     false   0           0         0             0
   0     false  false  false  0     0     0     0
   0     false  false  false  0     0     0     0
   0     false  false  false  0     0     0     0
   0     false  false  false  0     0     0     0
   0     false  false  false  0     0     0     0
   0     true  true  false  3     0     2     0
   

3. Verify that the broker is in maintenance mode:

   rpk cluster maintenance status
   Expected output

   NODE-ID  DRAINING  FINISHED  ERRORS  PARTITIONS  ELIGIBLE  TRANSFERRING  FAILED
   0        true      true      false   3           0         2             0
   1        false     false     false   0           0         0             0
   2        false     false     false   0           0         0             0

4. Validate again the health of the cluster:

   rpk cluster health
   Expected output

   CLUSTER HEALTH OVERVIEW
   =======================
   Healthy:                     true
   Controller ID:               0
   All nodes:                   [0 1 2]
   Nodes down:                  []
   Leaderless partitions:       []
   Under-replicated partitions: []
   note

   You can also evaluate external metrics to determine cluster health. If the cluster has any issues, take the broker out of maintenance mode by running the following command before proceeding with other operations, such as decommissioning or retrying the rolling upgrade:

   rpk cluster maintenance disable <node-id>

Upgrade your version

Linux

For Linux distributions, the process changes according to the distribution:

Fedora/RedHat

On the terminal, run:

sudo yum update redpanda

Debian/Ubuntu

On the terminal, run:

sudo apt update
sudo apt install redpanda

Docker

caution

Running Redpanda directly on Docker is not supported for production usage. This platform should only be used for testing.

To perform an upgrade you must replace the current image with a new one.

First, check which image is currently running on your Docker:

docker ps

Stop and remove Redpanda's container(s):

docker stop <container_id>
...
docker rm <container_id>

Remove current images:

docker rmi <image_id>

Pull the desired Redpanda's version, or adjust the setting to latest in the version tag:

docker pull docker.redpanda.com/redpandadata/redpanda:<version>

After it completes, restart the cluster:

docker restart <container_name>

For more information, see Redpanda Quickstart for Docker.

macOS

If you previously installed Redpanda with brew, run:

brew upgrade redpanda-data/tap/redpanda

For installations from binary files, download the preferred version from the release list and then overwrite the current rpk file in the installed location.

Check metrics

Check the following metrics before continuing with the upgrade:

Metric	Description
redpanda_kafka_under_replicated_replicas	If this shows any non-zero value, then replication cannot catch up, and the upgrade should be paused.
redpanda_cluster_unavailable_partitions	Before restart, wait for this to show zero unavailable partitions.
redpanda_kafka_request_bytes_total	Before restart, the produce and consume rate for each broker should recover to the pre-upgrade value.
redpanda_kafka_request_latency_seconds	Before restart, the p99 histogram should recover to the pre-upgrade value.
redpanda_rpc_request_latency_seconds	Before restart, the p99 histogram should recover to the pre-upgrade value.
redpanda_cpu_busy_seconds_total	Check the CPU utilization. The derivative gives you a 0.0-1.0 value for how much time the core was busy in a given second.

Restart broker

Restart the broker's Redpanda service with rpk redpanda stop, then rpk redpanda start.

Disable maintenance mode

After you've successfully upgraded the broker:

1. Take the broker out of maintenance mode:

   rpk cluster maintenance disable <node-id>
   Expected output

   Successfully disabled maintenance mode for node 0

2. Ensure that the broker is no longer in maintenance mode:

   rpk cluster maintenance status
   Expected output

   NODE-ID  DRAINING  FINISHED  ERRORS  PARTITIONS  ELIGIBLE  TRANSFERRING  FAILED
   0        false     false     false   0           0         0             0
   1        false     false     false   0           0         0             0
   2        false     false     false   0           0         0             0

Post-upgrade tasks

To verify that the cluster is running properly, run:

rpk cluster health

To view additional information about your brokers, run:

rpk redpanda admin brokers list

Suggested reading

To set up a real-time dashboard to monitor your cluster health, see Monitor Redpanda.