Debug Bundles Overview

A debug bundle is a collection of diagnostic data from your Redpanda cluster. It includes logs, metrics, and other critical information that helps diagnose and troubleshoot cluster issues.

Debug bundles are designed to:

  • Provide a consolidated view of cluster health and activity.

  • Help identify and resolve issues such as crashes, resource bottlenecks, or misconfigurations.

  • Facilitate communication with Redpanda support by enabling you to share detailed diagnostic data.

Tools for managing debug bundles

You can create and manage debug bundles using the following tools:

  • Redpanda Console: A user-friendly web UI for generating and downloading debug bundles without needing direct broker access.

  • rpk: A command-line tool that provides options for generating bundles either locally or remotely.

Contents of the debug bundle

The files and directories in the debug bundle differ depending on the environment in which Redpanda is running.

  • Linux

  • Kubernetes

File or Directory Description

/admin

Cluster and broker configurations, cluster health data, and license key information.

/controller

Binary-encoded replicated logs that contain the history of configuration changes as well as internal settings.
Redpanda can replay the events that took place in the cluster to arrive at a similar state.

data-dir.txt

Metadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed.

kafka.json

Kafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits.

redpanda.log

Redpanda logs for the broker.
If --logs-since is passed, only the logs within the given timeframe are included.

/metrics

Prometheus metrics from both the /metrics endpoint and the public_metrics endpoint.

/proc

CPU details of the broker on which the rpk debug bundle command was executed.
The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores.

redpanda.yaml

The Redpanda configuration file of the broker on which the rpk debug bundle command was executed.
Sensitive data is removed and replaced with (REDACTED).

resource-usage.json

Redpanda resource usage data, such as CPU usage and free memory available.

/utils

Data from the node on which the broker is running. This directory includes:

  • du.txt: The disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command.

  • ntp.txt: The NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed.

  • uname.txt: System information, such as the kernel version, hostname, and architecture, as output by the uname command.

  • dig.txt: The DNS resolution information for the node, as output by the dig command.

  • dmidecode.txt: System hardware information from the node, as output by the the dmidecode command. Requires root privileges.

  • free.txt: The amount of free and used memory on the node, as output by the free command.

  • ip.txt: Network interface information, including IP addresses and network configuration, as output by the ip command.

  • lspci.txt: Information about PCI devices on the node, as output by the lspci command.

  • ss.txt: Active socket connections, as output by the ss command, showing network connections, listening ports, and more.

  • sysctl.txt: Kernel parameters of the system, as output by the sysctl command.

  • top.txt: The top processes by CPU and memory usage, as output by the top command.

  • vmstat.txt: Virtual memory statistics, including CPU usage, memory, and IO operations, as output by the vmstat command.
File or Directory Description

/admin

Cluster and broker configurations, cluster health data, and license key information.

/controller

Binary-encoded replicated logs that contain the history of configuration changes as well as internal settings.
Redpanda can replay the events that took place in the cluster to arrive at a similar state.

data-dir.txt

Metadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed.

kafka.json

Kafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits.

redpanda.log

Redpanda logs for the broker.
If --logs-since is passed, only the logs within the given timeframe are included.

/metrics

Prometheus metrics from both the /metrics endpoint and the public_metrics endpoint.

/proc

CPU details of the broker on which the rpk debug bundle command was executed.
The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores.

redpanda.yaml

The Redpanda configuration file of the broker on which the rpk debug bundle command was executed.
Sensitive data is removed and replaced with (REDACTED).

resource-usage.json

Redpanda resource usage data, such as CPU usage and free memory available.

/utils

Data from the node on which the broker is running. This directory includes:

  • du.txt: The disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command.

  • ntp.txt: The NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed.

  • uname.txt: System information, such as the kernel version, hostname, and architecture, as output by the uname command.

  • dig.txt: The DNS resolution information for the node, as output by the dig command.

  • dmidecode.txt: System hardware information from the node, as output by the the dmidecode command. Requires root privileges.

  • free.txt: The amount of free and used memory on the node, as output by the free command.

  • ip.txt: Network interface information, including IP addresses and network configuration, as output by the ip command.

  • lspci.txt: Information about PCI devices on the node, as output by the lspci command.

  • ss.txt: Active socket connections, as output by the ss command, showing network connections, listening ports, and more.

  • sysctl.txt: Kernel parameters of the system, as output by the sysctl command.

  • top.txt: The top processes by CPU and memory usage, as output by the top command.

  • vmstat.txt: Virtual memory statistics, including CPU usage, memory, and IO operations, as output by the vmstat command.

In addition to the preceding files and directories, the debug bundle also includes up to 50 of the most recent stack traces from the root cause of a crash loop. Logs include:

  • Cluster UUID

  • Node ID

  • Timestamp

  • Stack trace logs

Next steps