Docs Self-Managed Manage Recovery Mode You are viewing the Self-Managed v24.3 beta documentation. We welcome your feedback at the Redpanda Community Slack #beta-feedback channel. To view the latest available version of the docs, see v24.2. Recovery Mode Recovery mode allows you to repair and restore a failed cluster that cannot start normally due to issues such as system crashes or out-of-memory (OOM) errors. In recovery mode, Redpanda limits functionality to cluster configuration changes and other manual administrative actions so that you can repair the cluster. Enabled functionality In recovery mode, Redpanda enables the following functionality so that you can repair the cluster: Kafka API Modify topic properties Delete topics Add and remove access control lists (ACLs) Edit consumer group metadata Admin API Edit cluster configuration properties Add and remove users Add new brokers to the cluster Delete WASM transforms Disabled functionality In recovery mode, Redpanda disables the following functionality to provide a more stable environment for troubleshooting issues and restoring the cluster to a usable state. The following APIs are disabled because some connections, especially malicious ones, can disrupt availability for all users, including admin users: Kafka API (fetch and produce requests) HTTP Proxy Schema Registry The following node-wide and cluster-wide processes are disabled as they may disrupt recovery operations: Partition and leader balancers Tiered Storage housekeeping Tiered Storage cache management Compaction Redpanda does not load user-managed partitions on disk to prevent triggering partition leadership elections and replication that may occur on startup. Prerequisites You must have the following: rpk installed. Local access to machines running Redpanda. Start Redpanda in recovery mode A broker can only enter recovery mode as it starts up, and not while it is already running. You first set the broker configuration property to enable recovery mode, and then do a broker restart. Run the rpk redpanda mode recovery command to set the recovery_mode_enabled broker configuration property to true. rpk redpanda mode recovery Enable recovery mode for all brokers. Although, you can start a mixed-mode cluster, where some brokers are in recovery mode while others are not, it’s not recommended. Restart the brokers. Check whether the cluster has entered recovery mode: rpk cluster health You should see a list of brokers that are in recovery mode. For example: CLUSTER HEALTH OVERVIEW ======================= Healthy: true Unhealthy reasons: [] Controller ID: 0 All nodes: [0 1 2] Nodes down: [] Nodes in recovery mode: [0 1 2] Leaderless partitions (0): [] Under-replicated partitions (0): [] In recovery mode, all private Redpanda topics such as __consumer_offsets are accessible. Data in user-created topics is not available, but you can still manage the metadata for these topics. Exit recovery mode Exit recovery mode by running one of the following commands: To exit into developer mode: rpk redpanda mode developer To exit into production mode: rpk redpanda mode production Disable partitions Problems that prevent normal cluster startup may be isolated to certain partitions or topics. You can use rpk or the Admin API to disable these partitions at the topic level, or individual partition level. A disabled partition or topic returns a Replica Not Available error code for Kafka API requests. To disable a partition, you need a healthy controller in the cluster, so you must start the cluster in recovery mode if a problematic partition is affecting cluster startup. If you disable a partition while in recovery mode, starting Redpanda again in non-recovery mode leaves the partition in a deactivated state. You must explicitly re-enable the partition. You can also disable a partition outside of recovery mode, if the issue is localized to the partition and does not interfere with cluster startup. The following examples show you how to use the Admin API to enable or disable partitions. The examples are based on the assumption that the Admin API port is 9644. Use kafka as the partition-namespace when making API calls to manage partitions in user topics. Disable a specific partition of a topic rpk Curl rpk cluster partitions disable <topic-name> --partitions <comma-delimited-partition-id> curl -X POST -d '{"disabled": true}' http://localhost:9644/cluster/partitions/<partition-namespace>/<topic-name>/<partition-id> Enable a specific partition of a topic rpk Curl rpk cluster partitions enable <topic-name> --partitions <comma-delimited-partition-id> curl -X POST -d '{"disabled": false}' http://localhost:9644/cluster/partitions/<partition-namespace>/<topic-name>/<partition-id> Disable all partitions of a specific topic rpk Curl rpk cluster partitions disable <topic-name> --all curl -X POST -d '{"disabled": true}' http://localhost:9644/cluster/partitions/<partition-namespace>/<topic-name> Enable all partitions of a specific topic rpk Curl rpk cluster partitions enable <topic-name> --all curl -X POST -d '{"disabled": false}' http://localhost:9644/cluster/partitions/<partition-namespace>/<topic-name> List all disabled partitions rpk Curl rpk cluster partitions list --all --disabled-only curl http://localhost:9644/cluster/partitions?disabled=true List all disabled partitions of a specific topic rpk Curl rpk cluster partitions list <topic-names> --disabled-only curl http://localhost:9644/cluster/partitions/<partition-namespace>/<topic-name>?disabled=true Suggested reading Admin API Perform a Rolling Restart Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Labs Rack Awareness