Migrate a Topic to Cloud Topics or Tiered Storage v2

Storage mode migration is a beta feature. It is not supported for production deployments. Beta features are available for users to test and provide feedback.

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

Starting in Redpanda v26.3, you can move an existing topic to the storage mode that fits its workload, without recreating the topic or stopping its producers and consumers. To compact the entire partition in object storage and remove tombstones, which Tiered Storage v1 can’t do, migrate to Tiered Storage v2 (beta). To lower storage costs for workloads that can accept higher latency, migrate to Cloud Topics.

You can migrate local, unset, and Tiered Storage v1 topics. Each topic keeps its name, partitions, offsets, and consumer group positions.

After reading this page, you will be able to:

  • Enable storage mode migration and migrate an existing topic to Cloud Topics or Tiered Storage v2

  • Monitor a migration and retry partitions whose migration stalls

  • Manage the Tiered Storage v1 data that a migration leaves behind

How migration works

When you change a topic’s storage mode to cloud or tiered_v2, the topic’s storage mode changes to the target immediately. Each partition then migrates in the background:

  1. Each partition keeps serving reads and writes in its original storage mode. Meanwhile, Redpanda records the partition’s existing Tiered Storage v1 data in the target storage mode. Redpanda references this data in place and doesn’t copy it. A partition of a local topic has no Tiered Storage v1 data to record, so it goes straight to the next step.

  2. The partition then cuts over to the target storage mode. New data is written in the target storage mode. Reads of older data use the referenced Tiered Storage v1 data, which ages out under the topic’s retention settings. Data that was only on local disk, such as the data of a local topic, starts copying to object storage.

  3. The migration is complete when every partition has cut over.

Partitions that you add to the topic during a migration are created in the target storage mode.

Prerequisites

  • All brokers run Redpanda v26.3 or later, and the topic_storage_mode_migration cluster feature is active. The feature activates when the upgrade to v26.3 is finalized. See Defer Upgrade Finalization.

  • Object storage is configured for the cluster.

  • A valid Enterprise license. Without one, requests that make a topic a Cloud Topic or Tiered Storage v2 topic are rejected.

  • To migrate to Tiered Storage v2, all brokers must also meet the Tiered Storage v2 requirements.

  • Superuser access to the Admin API, to monitor migration progress and retry stalled partitions. The rpk topic storage mode-migration commands use the Admin API.

Limitations

  • You can’t migrate a remote read replica topic directly. Instead, a remote read replica topic migrates along with its source topic, whether or not migration is enabled on the remote read replica cluster.

  • You can’t migrate a shadow topic directly, because Redpanda rejects changes to a shadow topic’s properties. Instead, a shadow topic follows the migration of its source topic. When you migrate a source topic, its shadow topic migrates too, if migration is enabled on the shadow cluster. If it isn’t, the shadow topic keeps its storage mode while replication continues, and it migrates after you enable migration on the shadow cluster.

  • You can’t cancel a migration or return a migrated topic to local or Tiered Storage v1. The only way back is the emergency procedure in Back out of a failed migration, which loses data. You can still convert a topic between cloud and tiered_v2, even during the migration.

  • Redpanda migrates at most topic_storage_mode_migration_max_concurrent_partitions partitions at a time across all topics (default: 128). Partitions over the limit wait until earlier partitions finish.

  • Topics listed in kafka_nodelete_topics or kafka_noproduce_topics, such as _schemas, can’t be migrated while they’re in those lists, because Kafka clients can’t alter their configuration.

Compacted topics, transactional topics, and Iceberg topics can be migrated. While a partition migrates, Cloud Topics compaction skips it until the partition cuts over. Transactions that are open when the migration starts complete normally, and Iceberg translation continues without gaps.

Enable migration

Migration is disabled by default. To enable it, set the enable_topic_storage_mode_migration cluster property to true:

rpk cluster config set enable_topic_storage_mode_migration true

This property doesn’t require a restart. If the topic_storage_mode_migration cluster feature isn’t active yet, the request is rejected with Requires the topic_storage_mode_migration feature to be active.

Optionally, change how many partitions can migrate at the same time with the topic_storage_mode_migration_max_concurrent_partitions cluster property. Lower values reduce the load that migrations place on the cluster, at the cost of longer migrations.

rpk cluster config set topic_storage_mode_migration_max_concurrent_partitions 64

Migrate a topic

To start a migration, change the topic’s storage mode with rpk topic alter-config.

To migrate a topic to Cloud Topics:

rpk topic alter-config <topic-name> --set redpanda.storage.mode=cloud

To migrate a topic to Tiered Storage v2 (beta), set the redpanda.storage.mode.impl topic property:

rpk topic alter-config <topic-name> --set redpanda.storage.mode.impl=tiered_v2

Setting redpanda.storage.mode.impl names the exact Tiered Storage version, regardless of the cluster default. If you set redpanda.storage.mode=tiered instead, the tiered value resolves to the version selected by the default_redpanda_storage_mode_tiered_impl cluster property. For a local topic, it starts a migration only when that property is tiered_v2. With the default (tiered_v1), setting tiered on a local topic enables Tiered Storage v1 instead of migrating the topic. Setting tiered on a Tiered Storage v1 topic has no effect, so to migrate a Tiered Storage v1 topic, set redpanda.storage.mode.impl=tiered_v2.

If you set both properties in the same request, their values must agree.

Because a migration can’t be reversed, rpk topic alter-config asks you to confirm any change that migrates a topic. The prompt lists the topics that the change migrates:

Warning: this change migrates the storage mode of topics [<topic-name>] to cloud or tiered_v2. A migration cannot be reverted. Abort?

Answer no to continue with the migration. To skip the prompt, for example in scripts, add the --no-confirm flag:

rpk topic alter-config <topic-name> --set redpanda.storage.mode=cloud --no-confirm

To confirm that the migration started, check the topic’s storage mode:

rpk topic describe <topic-name> -c | grep redpanda.storage.mode

The redpanda.storage.mode and redpanda.storage.mode.impl properties report the target storage mode as soon as the migration starts, even while partitions are still migrating. To see the progress of the migration, see Monitor migration progress.

If migration isn’t enabled, or the topic can’t be migrated, the request is rejected with an error such as:

Cannot alter redpanda.storage.mode from local to cloud - this transition is not permitted

For all permitted storage mode changes, see Change the storage mode.

Migrate Tiered Storage v1 topics to Tiered Storage v2

To migrate several Tiered Storage v1 topics, or every Tiered Storage v1 topic in the cluster, use rpk topic storage mode-migration migrate-tiered. The command sets redpanda.storage.mode.impl to tiered_v2 on each topic it selects.

To migrate specific topics:

rpk topic storage mode-migration migrate-tiered --topics <topic-1>,<topic-2>

To migrate every Tiered Storage v1 topic in the cluster:

rpk topic storage mode-migration migrate-tiered --all

The command selects a topic when all of the following are true:

  • Its redpanda.storage.mode.impl is tiered_v1, or it is unset and redpanda.remote.write is enabled.

  • Its partitions upload to object storage.

  • It isn’t a remote read replica.

The command skips all other topics, and reports the topics it skips. It also skips topics in kafka_nodelete_topics or kafka_noproduce_topics. To migrate one of these topics, remove it from the list, then pass it with --topics.

The --all flag also implies --update-default, which changes the cluster defaults so that new tiered topics use Tiered Storage v2:

  • It sets default_redpanda_storage_mode_tiered_impl to tiered_v2.

  • If default_redpanda_storage_mode is unset and cloud_storage_enable_remote_write is enabled, it also sets default_redpanda_storage_mode to tiered.

The command waits for every broker to apply the new defaults before it migrates any topic. It fails if a broker doesn’t apply them within 30 seconds. To keep the cluster defaults, add --update-default=false. To change the defaults when you migrate specific topics, add --update-default to a --topics command.

Before it changes anything, the command checks that cloud_storage_enabled is true, that the topic_storage_mode_migration cluster feature is active, and that enable_topic_storage_mode_migration is true. The command returns after each migration has started. It exits with a non-zero status if any topic fails to start migrating. To track progress, see Monitor migration progress.

Monitor migration progress

To check the progress of migrations, run rpk topic storage mode-migration describe. With no topic names, the command describes every topic that is migrating or has migrated.

rpk topic storage mode-migration describe <topic-name>

For each topic, the command prints:

Column Description

STATUS

none (the topic has no migration), migrating (a migration is in progress), or complete (every partition has migrated).

MIGRATED-FROM

The redpanda.storage.mode.impl value that the topic migrated from, such as local or tiered_v1, or - if the topic has never migrated.

PARTITIONS-COMPLETE

The number of partitions that have cut over to the target storage mode.

PARTITIONS-ADMITTED

The number of partitions allowed to start migrating so far. At most topic_storage_mode_migration_max_concurrent_partitions partitions migrate at the same time across the cluster, and more are admitted as earlier ones finish.

PARTITIONS-TO-MIGRATE

The number of partitions that were in the topic when the migration started. Partitions added during the migration are created in the target storage mode and aren’t counted.

BACKED-OFF-PARTITIONS

The IDs of partitions whose migration stalled and stopped, or - if there are none. See Retry stalled partitions.

To print the status in JSON or YAML, add --format json or --format yaml.

Redpanda records progress about every 10 seconds, so the status can trail the migration by that much.

You can also call the GetTopicStorageModeMigrationStatus Admin API v2 remote procedure call (RPC) on the admin port (default 9644). This call requires superuser privileges.

curl -s -X POST \
  http://<broker>:9644/redpanda.core.admin.v2.TopicService/GetTopicStorageModeMigrationStatus \
  -H 'Content-Type: application/json' -d '{"topic": "<topic-name>"}'

The response includes the status (TOPIC_STORAGE_MODE_MIGRATION_STATUS_NONE, TOPIC_STORAGE_MODE_MIGRATION_STATUS_MIGRATING, or TOPIC_STORAGE_MODE_MIGRATION_STATUS_COMPLETE), migrated_from, partitions_complete, partitions_admitted, partitions_to_migrate, and backed_off_partitions fields, which match the rpk columns. If the topic doesn’t exist, the call returns a not-found error.

You can also monitor migrations with the following public metrics:

Metric Description

redpanda_cloud_topics_migration_partitions

Number of migrating partitions per topic, labeled by redpanda_state (not_admitted, migrating, or complete).

redpanda_cloud_topics_migration_partitions_in_flight

Number of partitions migrating across the cluster.

redpanda_cloud_topics_migration_topics_migrating

Number of topics with a migration in progress.

redpanda_cloud_topics_migration_topics_complete_retaining

Number of topics whose partitions have all migrated, but whose Tiered Storage v1 data is still retained. See Manage Tiered Storage v1 data during migration.

Redpanda also logs a warning, at most every 10 minutes, for a partition in any of these states:

  • It has made no progress for 10 minutes.

  • It has had no leader for 10 minutes.

  • It has been migrating for 6 hours.

Retry stalled partitions

While a partition migrates, it pauses its uploads to object storage. If a partition makes no progress for the time set by the topic_storage_mode_migration_error_backoff_timeout cluster property (default: 1 hour), its migration stops, or backs off. A partition that backs off:

  • Keeps its original storage mode and resumes its uploads to object storage.

  • Stops counting toward topic_storage_mode_migration_max_concurrent_partitions, so partitions of other topics can take its place.

  • Blocks the rest of its topic’s migration. No more of the topic’s partitions start migrating until you retry the partition. Redpanda doesn’t retry it automatically.

A partition that has already cut over to the target storage mode never backs off.

To find partitions that backed off, check the BACKED-OFF-PARTITIONS column of rpk topic storage mode-migration describe, or the backed_off_partitions field of GetTopicStorageModeMigrationStatus. The internal metric vectorized_cloud_topics_migration_partitions_backed_off reports the number of partitions led by a broker whose migration backed off.

After you fix the cause, retry the migration of every backed-off partition of a topic:

rpk topic storage mode-migration retry <topic-name>

To retry only specific partitions, add -p (--partitions) with a comma-separated list of partition IDs:

rpk topic storage mode-migration retry <topic-name> -p 0,3

You can also call the RetryPartitionStorageModeMigration Admin API v2 RPC, one call for each partition. This call requires superuser privileges.

curl -s -X POST \
  http://<broker>:9644/redpanda.core.admin.v2.TopicService/RetryPartitionStorageModeMigration \
  -H 'Content-Type: application/json' -d '{"topic": "<topic-name>", "partition": <partition-id>}'

A retried partition starts its migration over and discards the progress of the previous attempt. A retry can temporarily take the number of migrating partitions above topic_storage_mode_migration_max_concurrent_partitions until enough partitions finish.

A retry fails if the partition hasn’t backed off, including a partition that isn’t migrating. It also fails if the partition’s leader can’t be reached or the partition is moving between brokers. In that case, run the command again later. A partition whose leader doesn’t respond isn’t listed as backed off, so rpk topic storage mode-migration retry without -p doesn’t retry it.

Manage Tiered Storage v1 data during migration

By default, Redpanda keeps a topic’s Tiered Storage v1 data in object storage until every partition of the topic has migrated. A migration can’t be undone. To back out of a failed migration, you delete the topic and recover it from this retained data. You can back out only while Redpanda keeps the data. For example, a migration can fail because of a software bug or corrupted Tiered Storage v1 metadata. See Back out of a failed migration. The cloud_migration_retain_tsv1_segments_until_migration_complete cluster property controls this behavior (default: true). The property applies only to partitions that start migrating after you enable it.

While the data is retained:

  • The migration reports the migrating status until every partition has migrated. The redpanda_cloud_topics_migration_topics_complete_retaining metric counts the topic when all partitions have migrated but retention hasn’t cleared yet.

  • Some Tiered Storage v1 data is never deleted from object storage. This is data that a partition stops using, for example because it falls out of retention, during a short window. The window opens when the partition finishes migrating and closes when every partition of the topic has migrated. The data stays in object storage after the migration completes and continues to accrue storage costs. Redpanda doesn’t currently provide a way to identify or remove this data. If you don’t need a way to back out of a migration, set cloud_migration_retain_tsv1_segments_until_migration_complete to false before you start the migration.

If you set cloud_migration_retain_tsv1_segments_until_migration_complete to false, Redpanda deletes Tiered Storage v1 data as the migrated topic stops using it. You then can’t back out of the migration.

Back out of a failed migration

If a migration fails and you can’t fix it, you can back out of it while the topic’s Tiered Storage v1 data is retained. Backing out loses data, so use it only as an emergency measure.

To back out, delete the topic. Redpanda leaves its retained Tiered Storage v1 data in object storage. Then recreate the topic as a Tiered Storage v1 topic with redpanda.remote.recovery set to true. See Topic Recovery.

Recovery restores the topic to its state before the migration: only the data that each partition had in object storage when it started migrating. Data written after a partition started migrating is lost.

Whole cluster restore during migration

If you restore a cluster that had a migration in progress, each partition is restored in the storage mode it had before the restore. A partition that had finished migrating is restored in the target storage mode, and the other partitions are restored in their original storage mode.

The topic’s migration then restarts. Redpanda checks each partition to confirm whether it already migrated, and continues the migration from the first partition that hadn’t. While it checks, rpk topic storage mode-migration describe reports the migrating status. A topic whose partitions had all migrated reports migrating until Redpanda has checked every partition.