Mountable Topics

For topics with Tiered Storage enabled, you can unmount a topic to safely detach it from a cluster and keep the topic data in the cluster’s object storage bucket or container. You can mount the detached topic to either the same origin cluster, or a different one. This allows you to hibernate a topic and free up system resources taken up by the topic, or migrate a topic to a different cluster.

Prerequisites

This feature requires an Enterprise license for self-managed deployments. To upgrade, contact Redpanda sales.

  1. Install rpk, or ensure that you have access to the Admin API.

  2. Enable Tiered Storage for specific topics, or for the entire cluster (all topics).

Unmount a topic from a cluster to object storage

When you unmount a topic, all incoming writes to the topic are blocked as Redpanda unmounts the topic from the cluster to object storage. Producers and consumers of the topic receive a message in the protocol replies indicating that the topic is no longer available:

  • Produce requests receive an invalid_topic_exception or resource_is_being_migrated response from the broker.

  • Consume requests receive an invalid_topic_exception response from the broker.

An unmounted topic in object storage is detached from all clusters. The original cluster releases ownership of the topic.

The unmounted topic is deleted in the source cluster, but can be mounted back again from object storage.

  • rpk

  • Admin API

In your cluster, run this command to unmount a topic to object storage:

rpk cluster storage unmount <namespace>/<topic-name>

To unmount topics from a cluster using the Admin API, make a POST request to the /v1/topics/unmount endpoint. Specify the names of the desired topics in the request body:

curl -X POST http://localhost:9644/v1/topics/unmount -d {
  "topics": [
    {
      "topic": "<topic-1-name>"
    },
    {
      "topic": "<topic-2-name>"
    },
    {
      "topic": "<topic-3-name>"
    }
  ]
}

You may optionally include the topic namespace (ns). Only kafka is supported.

You can use the ID returned by the command to monitor the progress of the unmount operation using rpk or the Admin API.

Mount a topic to a cluster

  • rpk

  • Admin API

  1. In your target cluster, run this command to list the topics that are available to mount from object storage:

    rpk cluster storage list-mountable

    The command output returns a LOCATION value in the format <topic-name>/<cluster-uuid>/<initial-revision>. Redpanda assigns an initial-revision number to a topic upon creation.

    The location value uniquely identifies a topic in object storage if multiple topics had the same name when they were unmounted from different origin clusters. For example:

    TOPIC      NAMESPACE  LOCATION
testtopic  kafka      testtopic/67f5505a-32f3-4677-bcad-3c75a1a702a6/10

    You can use the location as the topic reference instead of just the topic name to uniquely identify a topic to mount in the next step.

  2. Mount a topic from object storage:

    rpk cluster storage mount <topic-reference>

    Replace <topic-reference> with the name of the topic to mount. If there are multiple topics wih the same name in object storage, you are required to use the location value from rpk cluster storage list-mountable to uniquely identify a topic.

    You can also specify a new name for the topic as you mount it to the target cluster:

    rpk cluster storage mount <topic-reference> --to <new-topic-name>

    You only use the new name for the topic in the target cluster. This name does not persist if you unmount this topic again. Redpanda keeps the original name in object storage if you remount the topic later.

  1. List the topics that are available to mount from object storage by making a GET request to the v1/topics/mountable endpoint.

    curl http://localhost:9644/v1/topics/mountable

    The response object contains an array of topics:

    "topics": [
  {
    "topic_location": "topic-1-name/<cluster-1-uuid>/<initial-revision>",
    "topic": "topic-1-name"
  },
  {
    "topic_location": "topic-2-name/<cluster-1-uuid>/<initial-revision>",
    "topic": "topic-2-name"
  }
]

    The topic_location is the unique topic location in object storage, in the format <topic-name>/<cluster-uuid>/<initial-revision>. Redpanda assigns the number initial-revision to a topic upon creation. You can use topic-location as the topic reference instead of just the topic name to identify a unique topic to mount in the next step.

  2. To mount topics to a target cluster using the Admin API, make a POST request to the /v1/topics/mount endpoint. Specify the names of the topics in the request body:

    curl -X POST http://localhost:9644/v1/topics/mount -d {
  "topics": [
    {
      "source_topic_reference": {"ns": "kafka", "topic": "<topic-1-name>/<cluster-1-uuid>/<initial-revision>"},
      "alias": {"ns": "kafka", "topic": "<new-topic-1-name>"}
    },
    {
      "source_topic_reference": {"ns": "kafka", "topic": "<topic-2-name>"}
    },
    {
      "source_topic_reference": {"ns": "kafka", "topic": "<topic-3-name>/<cluster-2-uuid>/<initial-revision>"},
      "alias": {"ns": "kafka", "topic": "<new-topic-3-name>"}
    }
  ]
}

    • ns is the topic namespace. This field is optional and only kafka is supported.

    • You may have multiple topics with the same name that are available to mount from object storage. This can happen if you have unmounted topics with this name from different clusters. To uniquely identify a source topic, use <topic-name>/<cluster-uuid>/<initial-revision> as the topic reference.

    • To rename a topic in the target cluster, use the optional alias object in the request body. In the example, you specify new names for topics 1 and 3 in the target cluster, but topic 2 retains its original name in the target cluster.

You can use the ID returned by the operation to monitor its progress using rpk or the Admin API.

When the mount operation is complete, the target cluster handles produce and consume workloads for the topics.

Monitor progress

  • rpk

  • Admin API

To list active mount and unmount operations, run the command:

rpk cluster storage list-mount

Issue a GET request to the /migrations endpoint to view the status of topic mount and unmount operations:

curl http://localhost:9644/v1/migrations

You can also retrieve the status of a specific operation by running the command:

  • rpk

  • Admin API

rpk cluster storage status-mount <migration-id>
curl http://localhost:9644/v1/migrations/<migration-id>

<migration-id> is the unique identifier of the operation. Redpanda returns this ID when you start a mount or unmount. You can also retrieve the ID by listing existing operations.

The response returns the IDs and state of existing mount and unmount operations ("migrations"):

State Unmount operation (outbound) Mount operation (inbound)

planned

Redpanda validates the mount or unmount operation definition.

preparing

Redpanda flushes topic data, including topic manifests, to the destination bucket or container in object storage.

Redpanda recreates the topics in a disabled state in the target cluster. The cluster allocates partitions but does not add log segments yet. Topic metadata is populated from the topic manifests found in object storage.

prepared

The operation is ready to execute. In this state, the cluster still accepts client reads and writes for the topics.

Topics exist in the cluster but clients do not yet have access to consume or produce.

executing

The cluster rejects client reads and writes for the topics. Redpanda uploads any remaining topic data that has not yet been copied to object storage. Uncommitted transactions involving the topic are aborted.

The target cluster checks that the topic to be mounted has not already been mounted in any cluster.

executed

All unmounted topic data from the cluster is available in object storage.

The target cluster has verified that the topic has not already been mounted.

cut_over

Redpanda deletes topic metadata from the cluster, and marks the data in object storage as available for mount operations.

The topic data in object storage is no longer available to mount to any clusters.

finished

The operation is complete.

The operation is complete. The target cluster starts to handle produce and consume requests.

canceling

Redpanda is in the process of canceling the mount or unmount operation.

cancelled

The mount or unmount operation is cancelled.

Cancel a mount or unmount operation

You can cancel a topic mount or unmount by running the command:

  • rpk

  • Admin API

rpk cluster storage cancel-mount <migration-id>
curl -X POST http://localhost:9644/v1/<migration-id>/?action=cancel

You cannot cancel mount and unmount operations in the following states:

  • planned (but you may still delete a planned mount or unmount)

  • cut_over

  • finished

  • canceling

  • cancelled

Additional considerations

Redpanda prevents you from mounting the same topic to multiple clusters at once. This ensures that multiple clusters don’t write to the same location in object storage and corrupt the topic.

If you attempt to mount a topic where the name matches a topic already in the target cluster, Redpanda fails the operation and emits a warning message in the logs.