Docs Self-Managed Troubleshoot Generate Debug Bundle Kubernetes You are viewing the Self-Managed v24.3 beta documentation. We welcome your feedback at the Redpanda Community Slack #beta-feedback channel. To view the latest available version of the docs, see v24.2. Generate a Debug Bundle in Kubernetes Use rpk or Redpanda Console to generate a debug bundle to diagnose issues yourself, or send it to the Redpanda support team to help resolve your issue. Generate a debug bundle with rpk To generate a debug bundle with rpk, you can run the rpk debug bundle command on each broker in the cluster. Create a ClusterRole to allow Redpanda to collect information from the Kubernetes API: Helm + Operator Helm redpanda-cluster.yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: serviceAccount: create: true rbac: enabled: true kubectl apply -f redpanda-cluster.yaml --namespace <namespace> You must deploy the Redpanda Operator with the --set rbac.createRPKBundleCRs=true flag to give it the required ClusterRoles. --values --set serviceaccount.yaml serviceAccount: create: true rbac: enabled: true helm upgrade --install redpanda redpanda/redpanda --namespace redpanda --create-namespace \ --values serviceaccount.yaml --reuse-values helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \ --set serviceAccount.create=true \ --set rbac.enabled=true If you aren’t using the Helm chart, you can create the ClusterRole manually: kubectl create clusterrolebinding redpanda --clusterrole=view --serviceaccount=redpanda:default Execute the rpk debug bundle command on a broker: kubectl exec -it --namespace <namespace> redpanda-0 -c redpanda -- rpk debug bundle --namespace <namespace> If you have an upload URL from the Redpanda support team, provide it in the --upload-url flag to upload your debug bundle to Redpanda. kubectl exec -it --namespace <namespace> redpanda-0 -c redpanda -- rpk debug bundle \ --upload-url <url> \ --namespace <namespace> Example output: Creating bundle file... Debug bundle saved to "/var/lib/redpanda/1675440652-bundle.zip" On your host machine, make a directory in which to save the debug bundle: mkdir debug-bundle Copy the debug bundle ZIP file to the debug-bundle directory on your host machine. Replace <bundle-name> with the name of your ZIP file. kubectl cp <namespace>/redpanda-0:/var/lib/redpanda/<bundle-name> debug-bundle/<bundle-name>.zip Unzip the file on your host machine. cd debug-bundle unzip <bundle-name>.zip Remove the debug bundle from the Redpanda broker: kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rm /var/lib/redpanda/<bundle-name>.zip When you’ve finished troubleshooting, remove the debug bundle from your host machine: rm -r debug-bundle For a description of the files and directories, see Contents of the debug bundle. Generate a debug bundle with Redpanda Console See Manage Debug Bundles in Redpanda Console. Inspect the debug bundle After downloading the debug bundle files, you can inspect the contents to debug your cluster. This section provides some useful data points to check while troubleshooting. Most files in the debug bundle are JSON files. To make it easier to read these files, this section uses jq. To install jq, see the jq downloads page. View the version of Redpanda on all brokers cat admin/brokers.json | jq '.[] | .version' Example output: "v24.3.1" "v24.3.1" "v24.3.1" View the maintenance status of all brokers cat admin/brokers.json | jq '.[] | .node_id, .maintenance_status' Example output 0 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } 1 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } 2 { "draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0 } View the cluster configuration cat admin/cluster_config.json | jq Example output { "abort_index_segment_size": 50000, "abort_timed_out_transactions_interval_ms": 10000, "admin_api_require_auth": false, "aggregate_metrics": false, "alter_topic_cfg_timeout_ms": 5000, "append_chunk_size": 16384, "auto_create_topics_enabled": false, "cloud_storage_access_key": null, "cloud_storage_api_endpoint": null, "cloud_storage_api_endpoint_port": 443, "cloud_storage_azure_container": null, "cloud_storage_azure_shared_key": null, "cloud_storage_azure_storage_account": null, "cloud_storage_bucket": null, ... "target_quota_byte_rate": 2147483648, "tm_sync_timeout_ms": 10000, "topic_fds_per_partition": 5, "topic_memory_per_partition": 1048576, "topic_partitions_per_shard": 1000, "topic_partitions_reserve_shard0": 2, "transaction_coordinator_cleanup_policy": "delete", "transaction_coordinator_delete_retention_ms": 604800000, "transaction_coordinator_log_segment_size": 1073741824, "transactional_id_expiration_ms": 604800000, "tx_log_stats_interval_s": 10, "tx_timeout_delay_ms": 1000, "wait_for_leader_timeout_ms": 5000, "zstd_decompress_workspace_bytes": 8388608 } Check Enterprise Edition license keys cat admin/license.json | jq Example output { "loaded": false, "license": { "format_version": 0, "org": "", "type": "", "expires": 0, "sha256": "" } } View metadata about the Redpanda data directory To check the size of the directories and look for anomalies: cat utils/du.txt Example output 33M /var/lib/redpanda/data/redpanda/kvstore/0_0 33M /var/lib/redpanda/data/redpanda/kvstore 33M /var/lib/redpanda/data/redpanda/controller/0_0 33M /var/lib/redpanda/data/redpanda/controller 65M /var/lib/redpanda/data/redpanda 65M /var/lib/redpanda/data To check the file permissions, file size, and last modification date of the files: cat data-dir.txt | jq Example output { "/var/lib/redpanda/data": { "size": "4.096kB", "mode": "dgrwxrwxrwx", "modified": "2023-02-02 15:21:12.430878371 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/config_cache.yaml": { "size": "340B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:22.434878593 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/pid.lock": { "size": "2B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:10.502878322 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.650878326 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.650878326 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller/0_0": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:12.346878368 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log": { "size": "4.096kB", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:32.450878771 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.590878324 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore/0_0": { "size": "4.096kB", "mode": "dgrwxr-xr-x", "modified": "2023-02-02 15:21:10.602878325 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log": { "size": "8.192kB", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:32.458878772 +0000 UTC", "user": "", "group": "redpanda" }, "/var/lib/redpanda/data/startup_log": { "size": "26B", "mode": "-rw-r--r--", "modified": "2023-02-02 15:21:10.510878323 +0000 UTC", "user": "", "group": "redpanda" } } View cluster metadata cat kafka.json | jq '.[0]' Example output { "Name": "metadata", "Response": { "Cluster": "redpanda.14a3f9b6-1c74-4ffd-806a-4ab48db78120", "Controller": 0, "Brokers": [ { "NodeID": 0, "Port": 9093, "Host": "redpanda-0.redpanda.<namespace>.svc.cluster.local.", "Rack": null }, { "NodeID": 1, "Port": 9093, "Host": "redpanda-1.redpanda.<namespace>.svc.cluster.local.", "Rack": null }, { "NodeID": 2, "Port": 9093, "Host": "redpanda-2.redpanda.<namespace>.svc.cluster.local.", "Rack": null } ], "Topics": {} }, "Error": null } View topic and broker configurations cat kafka.json | jq '.[1:]' Example output [ { "Name": "topic_configs", "Response": null, "Error": null }, { "Name": "broker_configs", "Response": [ { "Name": "0", "Configs": [ { "Key": "listeners", "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "kafka_api", "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094", "Source": "STATIC_BROKER_CONFIG" }, { "Key": "kafka_api", "Value": "plain://127.0.0.1:9092", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "advertised.listeners", "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "advertised_kafka_api", "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092", "Source": "STATIC_BROKER_CONFIG" }, { "Key": "advertised_kafka_api", "Value": "", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.segment.bytes", "Value": "134217728", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "log_segment_size", "Value": "134217728", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.retention.bytes", "Value": "18446744073709551615", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "retention_bytes", "Value": "18446744073709551615", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.retention.ms", "Value": "604800000", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "delete_retention_ms", "Value": "604800000", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "num.partitions", "Value": "1", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "default_topic_partitions", "Value": "1", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "default.replication.factor", "Value": "1", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "default_topic_replications", "Value": "1", "Source": "DEFAULT_CONFIG" } ] }, { "Key": "log.dirs", "Value": "/var/lib/redpanda/data", "Sensitive": false, "Source": "STATIC_BROKER_CONFIG", "Synonyms": [ { "Key": "data_directory", "Value": "/var/lib/redpanda/data", "Source": "STATIC_BROKER_CONFIG" } ] }, { "Key": "auto.create.topics.enable", "Value": "false", "Sensitive": false, "Source": "DEFAULT_CONFIG", "Synonyms": [ { "Key": "auto_create_topics_enabled", "Value": "false", "Source": "DEFAULT_CONFIG" } ] } ], "Err": null }, { "Name": "1", "Configs": [ ... ] ... }, { "Name": "1", "Configs": [ ... ] ... }, ], "Error": null }, { "Name": "log_start_offsets", "Response": {}, "Error": null }, { "Name": "last_stable_offsets", "Response": {}, "Error": null }, { "Name": "high_watermarks", "Response": {}, "Error": null }, { "Name": "groups", "Response": null, "Error": null } ] View the Redpanda logs cat logs/redpanda-0.txt # logs/redpanda-1.txt logs/redpanda-2.txt Check for clock drift cat utils/ntp.txt | jq Use the output to check for clock drift. For details about how NTP works, see the NTP documentation. Example output { "host": "pool.ntp.org", "roundTripTimeMs": 3, "remoteTimeUTC": "2023-02-02T15:22:51.763175934Z", "localTimeUTC": "2023-02-02T15:22:51.698044603Z", "precisionMs": 0, "offset": -458273 } View Kubernetes manifests tree k8s Example output k8s ├── configmaps.json ├── endpoints.json ├── events.json ├── limitranges.json ├── persistentvolumeclaims.json ├── pods.json ├── replicationcontrollers.json ├── resourcequotas.json ├── serviceaccounts.json └── services.json Contents of the debug bundle The debug bundle includes the following files and directories: Redpanda collects some data from the Kubernetes API. To communicate with the Kubernetes API, Redpanda requires a ClusterRole attached to the default ServiceAccount for the Pods. The files and directories that are generated only when the ClusterRole exists are labeled Requires ClusterRole. File or Directory Description /admin Cluster and broker configurations, cluster health data, and license key information. Requires ClusterRole. /controller Binary-encoded replicated logs that contain the history of configuration changes as well as internal settings. Redpanda can replay the events that took place in the cluster to arrive at a similar state. data-dir.txt Metadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed. /k8s Kubernetes manifests for all resources in the given Kubernetes namespace. Requires ClusterRole. kafka.json Kafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits. /logs Logs from the Pods that run Redpanda in the given Kubernetes namespace. If --logs-since is passed, only the logs within the given timeframe are included. Requires ClusterRole. /metrics Prometheus metrics from both the /metrics endpoint and the public_metrics endpoint. Requires ClusterRole. /proc CPU details of the broker on which the rpk debug bundle command was executed. The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores. redpanda.yaml The Redpanda configuration file of the broker on which the rpk debug bundle command was executed. Sensitive data is removed and replaced with (REDACTED). resource-usage.json Redpanda resource usage data, such as CPU usage and free memory available. /utils Data from the node on which the broker is running. This directory includes: du.txt: The disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command. ntp.txt: The NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed. uname.txt: System information, such as the kernel version, hostname, and architecture, as output by the uname command. Suggested reading rpk debug bundle jq manual Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Linux Redpanda Console