Docs Self-Managed Troubleshoot Generate Debug Bundle Redpanda Console Manage Debug Bundles in Redpanda Console This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales. If Redpanda Console has enterprise features enabled and it cannot find a valid license either in its local configuration or in the connected Redpanda cluster, it redirects you to the license expiration landing page, and all other access is restricted. Learn how to generate, download, and delete debug bundles in Redpanda Console for comprehensive cluster diagnostics. Prerequisites Redpanda Console must be connected to a Redpanda cluster and configured to connect to the Redpanda Admin API. Redpanda Console must be configured with an authentication provider and have at least one admin user. Generate a debug bundle You can generate a debug bundle for all brokers in the cluster and download it onto your local computer for inspection. Only one debug bundle can exist at a time. If you generate a new debug bundle, any existing bundle from a previous run will be automatically deleted. Log in as an admin user. Click Admin in the sidebar. This option is visible only when logged in as an admin user. Go to Debug bundle. Click Generate new. Wait until the process is complete. Click debug-bundle.zip to download the bundle on your local computer. Unzip the file to inspect the contents. Inspect the debug bundle After downloading the debug bundle files, you can inspect the contents to debug your cluster. This section provides some useful data points to check while troubleshooting. Most files in the debug bundle are JSON files. To make it easier to read these files, this section uses jq. To install jq, see the jq downloads page. View the version of Redpanda on all brokers cat admin/brokers.json | jq '.[] | .version' Example output: "v24.3.1" "v24.3.1" "v24.3.1" View the maintenance status of all brokers cat admin/brokers.json | jq '.[] | .node_id, .maintenance_status' Example output 0 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } 1 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } 2 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } View the cluster configuration cat admin/cluster_config.json | jq Example output { "abort_index_segment_size": 50000, "abort_timed_out_transactions_interval_ms": 10000, "admin_api_require_auth": false, "aggregate_metrics": false, "alter_topic_cfg_timeout_ms": 5000, "append_chunk_size": 16384, "auto_create_topics_enabled": false, "cloud_storage_access_key": null, "cloud_storage_api_endpoint": null, "cloud_storage_api_endpoint_port": 443, "cloud_storage_azure_container": null, "cloud_storage_azure_shared_key": null, "cloud_storage_azure_storage_account": null, "cloud_storage_bucket": null, ... "target_quota_byte_rate": 2147483648, "tm_sync_timeout_ms": 10000, "topic_fds_per_partition": 5, "topic_memory_per_partition": 1048576, "topic_partitions_per_shard": 1000, "topic_partitions_reserve_shard0": 2, "transaction_coordinator_cleanup_policy": "delete", "transaction_coordinator_delete_retention_ms": 604800000, "transaction_coordinator_log_segment_size": 1073741824, "transactional_id_expiration_ms": 604800000, "tx_log_stats_interval_s": 10, "tx_timeout_delay_ms": 1000, "wait_for_leader_timeout_ms": 5000, "zstd_decompress_workspace_bytes": 8388608 } Check Enterprise Edition license keys cat admin/license.json | jq Example output { "loaded": false, "license": { "format_version": 0, "org": "", "type": "", "expires": 0, "sha256": "" } } View metadata about the Redpanda data directory You can inspect the size of directories in the Redpanda data directory and identify anomalies using a pre-generated report. This is useful for troubleshooting issues such as imbalances in partition sizes or unexpected data growth. To check the size of the directories and look for anomalies: cat utils/du.txt The du.txt file provides information about the size of each directory. Anomalies to look for include: One partition of a topic being significantly larger than others. Directories containing more data than expected for a specific topic or partition. Example output 33M /var/lib/redpanda/data/redpanda/kvstore/0_0 33M /var/lib/redpanda/data/redpanda/kvstore 33M /var/lib/redpanda/data/redpanda/controller/0_0 33M /var/lib/redpanda/data/redpanda/controller 65M /var/lib/redpanda/data/redpanda 65M /var/lib/redpanda/data To check the file permissions, file size, and last modification date of the files: cat data-dir.txt | jq Example output { "/var/lib/redpanda/data": { "size": "4.096kB", "mode": "dgrwxrwxrwx", "modified": "2023-02-02 15:21:12.430878371 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/config_cache.yaml": { "size": "340B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:22.434878593 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/pid.lock": { "size": "2B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:10.502878322 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.650878326 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.650878326 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller/0_0": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:12.346878368 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log": { "size": "4.096kB", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:32.450878771 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.590878324 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore/0_0": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.602878325 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log": { "size": "8.192kB", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:32.458878772 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/startup_log": { "size": "26B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:10.510878323 +0000 UTC", "user": "", "group": "redpanda" } } View cluster metadata cat kafka.json | jq '.[0]' Example output { "Name": "metadata", "Response": { "Cluster": "redpanda.14a3f9b6-1c74-4ffd-806a-4ab48db78120", "Controller": 0, "Brokers": [ { "NodeID": 0, "Port": 9093, "Host": "redpanda-0.redpanda.<namespace>.svc.cluster.local.", "Rack": null }, { "NodeID": 1, "Port": 9093, "Host": "redpanda-1.redpanda.<namespace>.svc.cluster.local.", "Rack": null }, { "NodeID": 2, "Port": 9093, "Host": "redpanda-2.redpanda.<namespace>.svc.cluster.local.", "Rack": null } ], "Topics": {} }, "Error": null } View topic and broker configurations cat kafka.json | jq '.[1:]' Example output [ { "Name": "topic_configs", "Response": null, "Error": null }, { "Name": "broker_configs", "Response": [ { "Name": "0", "Configs": [ { "Key": "listeners", "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "kafka_api", "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094", "Source": "STATIC_BROKER_CONFIG" }, { "Key": "kafka_api", "Value": "plain://127.0.0.1:9092", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "advertised.listeners", "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "advertised_kafka_api", "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092", "Source": "STATIC_BROKER_CONFIG" }, { "Key": "advertised_kafka_api", "Value": "", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.segment.bytes", "Value": "134217728", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "log_segment_size", "Value": "134217728", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.retention.bytes", "Value": "18446744073709551615", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "retention_bytes", "Value": "18446744073709551615", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.retention.ms", "Value": "604800000", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "delete_retention_ms", "Value": "604800000", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "num.partitions", "Value": "1", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "default_topic_partitions", "Value": "1", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "default.replication.factor", "Value": "1", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "default_topic_replications", "Value": "1", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.dirs", "Value": "/var/lib/redpanda/data", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "data_directory", "Value": "/var/lib/redpanda/data", "Source": "STATIC_BROKER_CONFIG" } ] }, { "Key": "auto.create.topics.enable", "Value": "false", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "auto_create_topics_enabled", "Value": "false", "Source": "DEFAULT_CONFIG" } ] } ], "Err": null }, { "Name": "1", "Configs": [ ... ] ... }, { "Name": "1", "Configs": [ ... ] ... }, ], "Error": null }, { "Name": "log_start_offsets", "Response": {}, "Error": null }, { "Name": "last_stable_offsets", "Response": {}, "Error": null }, { "Name": "high_watermarks", "Response": {}, "Error": null }, { "Name": "groups", "Response": null, "Error": null } ] View the Redpanda logs cat redpanda.log Check for clock drift cat utils/ntp.txt | jq Use the output to check for clock drift. For details about how NTP works, see the NTP documentation. Example output { "host": "pool.ntp.org", "roundTripTimeMs": 3, "remoteTimeUTC": "2023-02-02T15:22:51.763175934Z", "localTimeUTC": "2023-02-02T15:22:51.698044603Z", "precisionMs": 0, "offset": -458273 } Delete a debug bundle To manually delete a debug bundle: Log in as an admin user. Click Admin in the sidebar. This option is visible only when logged in as an admin user. Go to Debug bundle. Click the trash icon next to debug-bundle.zip to delete the bundle. Next steps Cluster Diagnostics Error Messages and Solutions Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Kubernetes Resolve Errors