Docs Self-Managed Troubleshoot Debug Bundles Inspect Inspect a Debug Bundle After downloading the debug bundle files, you can inspect the contents to debug your cluster. This topic provides some useful data points to check while troubleshooting. Most files in the debug bundle are JSON files. To make it easier to read these files, this section uses jq. To install jq, see the jq downloads page. Prerequisites Generate a debug bundle. View the version of Redpanda on all brokers cat admin/brokers.json | jq '.[] | .version' Example output: "v24.3.1" "v24.3.1" "v24.3.1" View the maintenance status of all brokers cat admin/brokers.json | jq '.[] | .node_id, .maintenance_status' Example output 0 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } 1 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } 2 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } View the cluster configuration cat admin/cluster_config.json | jq Example output { "abort_index_segment_size": 50000, "abort_timed_out_transactions_interval_ms": 10000, "admin_api_require_auth": false, "aggregate_metrics": false, "alter_topic_cfg_timeout_ms": 5000, "append_chunk_size": 16384, "auto_create_topics_enabled": false, "cloud_storage_access_key": null, "cloud_storage_api_endpoint": null, "cloud_storage_api_endpoint_port": 443, "cloud_storage_azure_container": null, "cloud_storage_azure_shared_key": null, "cloud_storage_azure_storage_account": null, "cloud_storage_bucket": null, ... "target_quota_byte_rate": 2147483648, "tm_sync_timeout_ms": 10000, "topic_fds_per_partition": 5, "topic_memory_per_partition": 1048576, "topic_partitions_per_shard": 1000, "topic_partitions_reserve_shard0": 2, "transaction_coordinator_cleanup_policy": "delete", "transaction_coordinator_delete_retention_ms": 604800000, "transaction_coordinator_log_segment_size": 1073741824, "transactional_id_expiration_ms": 604800000, "tx_log_stats_interval_s": 10, "tx_timeout_delay_ms": 1000, "wait_for_leader_timeout_ms": 5000, "zstd_decompress_workspace_bytes": 8388608 } Check Enterprise Edition license keys cat admin/license.json | jq Example output { "loaded": false, "license": { "format_version": 0, "org": "", "type": "", "expires": 0, "sha256": "" } } View metadata about the Redpanda data directory You can inspect the size of directories in the Redpanda data directory and identify anomalies using a pre-generated report. This is useful for troubleshooting issues such as imbalances in partition sizes or unexpected data growth. To check the size of the directories and look for anomalies: cat utils/du.txt The du.txt file provides information about the size of each directory. Anomalies to look for include: One partition of a topic being significantly larger than others. Directories containing more data than expected for a specific topic or partition. Example output 33M /var/lib/redpanda/data/redpanda/kvstore/0_0 33M /var/lib/redpanda/data/redpanda/kvstore 33M /var/lib/redpanda/data/redpanda/controller/0_0 33M /var/lib/redpanda/data/redpanda/controller 65M /var/lib/redpanda/data/redpanda 65M /var/lib/redpanda/data To check the file permissions, file size, and last modification date of the files: cat data-dir.txt | jq Example output { "/var/lib/redpanda/data": { "size": "4.096kB", "mode": "dgrwxrwxrwx", "modified": "2023-02-02 15:21:12.430878371 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/config_cache.yaml": { "size": "340B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:22.434878593 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/pid.lock": { "size": "2B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:10.502878322 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.650878326 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.650878326 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller/0_0": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:12.346878368 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log": { "size": "4.096kB", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:32.450878771 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.590878324 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore/0_0": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.602878325 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log": { "size": "8.192kB", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:32.458878772 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/startup_log": { "size": "26B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:10.510878323 +0000 UTC", "user": "", "group": "redpanda" } } View cluster metadata cat kafka.json | jq '.[0]' Example output { "Name": "metadata", "Response": { "Cluster": "redpanda.14a3f9b6-1c74-4ffd-806a-4ab48db78120", "Controller": 0, "Brokers": [ { "NodeID": 0, "Port": 9093, "Host": "redpanda-0.redpanda.<namespace>.svc.cluster.local.", "Rack": null }, { "NodeID": 1, "Port": 9093, "Host": "redpanda-1.redpanda.<namespace>.svc.cluster.local.", "Rack": null }, { "NodeID": 2, "Port": 9093, "Host": "redpanda-2.redpanda.<namespace>.svc.cluster.local.", "Rack": null } ], "Topics": {} }, "Error": null } View topic and broker configurations cat kafka.json | jq '.[1:]' Example output [ { "Name": "topic_configs", "Response": null, "Error": null }, { "Name": "broker_configs", "Response": [ { "Name": "0", "Configs": [ { "Key": "listeners", "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "kafka_api", "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094", "Source": "STATIC_BROKER_CONFIG" }, { "Key": "kafka_api", "Value": "plain://127.0.0.1:9092", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "advertised.listeners", "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "advertised_kafka_api", "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092", "Source": "STATIC_BROKER_CONFIG" }, { "Key": "advertised_kafka_api", "Value": "", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.segment.bytes", "Value": "134217728", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "log_segment_size", "Value": "134217728", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.retention.bytes", "Value": "18446744073709551615", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "retention_bytes", "Value": "18446744073709551615", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.retention.ms", "Value": "604800000", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "delete_retention_ms", "Value": "604800000", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "num.partitions", "Value": "1", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "default_topic_partitions", "Value": "1", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "default.replication.factor", "Value": "1", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "default_topic_replications", "Value": "1", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.dirs", "Value": "/var/lib/redpanda/data", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "data_directory", "Value": "/var/lib/redpanda/data", "Source": "STATIC_BROKER_CONFIG" } ] }, { "Key": "auto.create.topics.enable", "Value": "false", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "auto_create_topics_enabled", "Value": "false", "Source": "DEFAULT_CONFIG" } ] } ], "Err": null }, { "Name": "1", "Configs": [ ... ] ... }, { "Name": "1", "Configs": [ ... ] ... }, ], "Error": null }, { "Name": "log_start_offsets", "Response": {}, "Error": null }, { "Name": "last_stable_offsets", "Response": {}, "Error": null }, { "Name": "high_watermarks", "Response": {}, "Error": null }, { "Name": "groups", "Response": null, "Error": null } ] View the Redpanda logs If your cluster is running in Kubernetes, inspect the logs/ directory: cat logs/redpanda-0.txt # logs/redpanda-1.txt logs/redpanda-2.txt Otherwise, inspect the redpanda.log file: cat redpanda.log Check for clock drift cat utils/ntp.txt | jq Use the output to check for clock drift. For details about how NTP works, see the NTP documentation. Example output { "host": "pool.ntp.org", "roundTripTimeMs": 3, "remoteTimeUTC": "2023-02-02T15:22:51.763175934Z", "localTimeUTC": "2023-02-02T15:22:51.698044603Z", "precisionMs": 0, "offset": -458273 } View Kubernetes manifests If your cluster is running in Kubernetes, you can view all manifest in the k8s/ directory: tree k8s Example output k8s ├── configmaps.json ├── endpoints.json ├── events.json ├── limitranges.json ├── persistentvolumeclaims.json ├── pods.json ├── replicationcontrollers.json ├── resourcequotas.json ├── serviceaccounts.json └── services.json Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Redpanda Console Resolve Errors