# Diagnostics Bundles in Kubernetes

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt)

---
title: Diagnostics Bundles in Kubernetes
latest-operator-version: v26.1.4
# EOL = End-of-Life (support lifecycle status)
page-is-nearing-eol: "false"
page-is-past-eol: "true"
page-eol-date: December 22, 2024
latest-console-tag: v3.7.3
latest-connect-version: 4.93.0
docname: kubernetes/troubleshooting/k-diagnostics-bundle
page-component-name: streaming
page-version: "23.3"
page-component-version: "23.3"
page-component-title: Streaming
page-relative-src-path: kubernetes/troubleshooting/k-diagnostics-bundle.adoc
page-edit-url: https://github.com/redpanda-data/docs/edit/v/23.3/modules/manage/pages/kubernetes/troubleshooting/k-diagnostics-bundle.adoc
description: Use the diagnostics bundle to debug issues yourself, or you can send it to the Redpanda support team to help resolve your issue.
page-git-created-date: "2024-01-04"
page-git-modified-date: "2024-02-26"
support-status: past end-of-life
---

<!-- Source: https://docs.redpanda.com/streaming/23.3/manage/kubernetes/troubleshooting/k-diagnostics-bundle.md -->

A diagnostics bundle is a ZIP file with data that can help debug and diagnose issues with a Redpanda cluster, a broker, or the machines on which the brokers are running. You can use this file to debug issues yourself, or you can send it to the Redpanda support team to help resolve your issue.

## [](#prerequisites)Prerequisites

Most files in the diagnostics bundle are JSON files. To make it easier to read these files, this guide uses jq. To install jq, see the [jq downloads page](https://stedolan.github.io/jq/download/).

These examples use the default values in Helm chart. If you’ve customized the Helm chart, you may need to provide custom values and/or flags.

## [](#generate-a-diagnostics-bundle)Generate a diagnostics bundle

1.  Create a ClusterRole to allow Redpanda to collect information from the Kubernetes API:

    ### Helm + Operator

    `redpanda-cluster.yaml`

    ```yaml
    apiVersion: cluster.redpanda.com/v1alpha1
    kind: Redpanda
    metadata:
      name: redpanda
    spec:
      chartRef: {}
      clusterSpec:
        serviceAccount:
          create: true
        rbac:
          enabled: true
    ```

    ```bash
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
    ```

    > 📝 **NOTE**
    >
    > You must deploy the Redpanda Operator with the `--set rbac.createRPKBundleCRs=true` flag to give it the required ClusterRoles.


    ### Helm


    #### --values

    `serviceaccount.yaml`

    ```yaml
    serviceAccount:
      create: true
    rbac:
      enabled: true
    ```

    ```bash
    helm upgrade --install redpanda redpanda/redpanda --namespace redpanda --create-namespace \
      --values serviceaccount.yaml --reuse-values
    ```


    #### --set

    ```bash
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
      --set serviceAccount.create=true \
      --set rbac.enabled=true
    ```

    > 💡 **TIP**
    >
    > If you aren’t using the Helm chart, you can create the ClusterRole manually:
    >
    > ```bash
    > kubectl create clusterrolebinding redpanda --clusterrole=view --serviceaccount=redpanda:default
    > ```

2.  Execute the [`rpk debug bundle`](https://docs.redpanda.com/streaming/23.3/reference/rpk/rpk-debug/rpk-debug-bundle/) command inside a Pod container that’s running a Redpanda broker.

    In this example, the command is executed on a Pod called `redpanda-0`.

    ```bash
    kubectl exec -it --namespace <namespace> redpanda-0 -c redpanda -- rpk debug bundle
    ```

    If you have an upload URL from the Redpanda support team, provide it in the `--upload-url` flag to upload your diagnostics bundle to Redpanda.

    ```bash
    kubectl exec -it --namespace <namespace> redpanda-0 -c redpanda -- rpk debug bundle \
      --upload-url <url> \
      --namespace <namespace>
    ```

    Example output:

    Creating bundle file...

    Debug bundle saved to "/var/lib/redpanda/1675440652-bundle.zip"

3.  On your host machine, make a directory in which to save the diagnostics bundle:

    ```bash
    mkdir diagnostics-bundle
    ```

4.  Copy the diagnostics bundle from the Pod to your host machine:

    Replace `<bundle-name>` with the name of your ZIP file.

    ```bash
    kubectl cp redpanda/redpanda-0:/var/lib/redpanda/<bundle-name> diagnostics-bundle/<bundle-name>
    ```

5.  Unzip the file on your host machine.

    ```bash
    cd diagnostics-bundle
    unzip <bundle-name>
    ```

6.  Remove the diagnostics bundle from the Redpanda container:

    ```bash
    kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rm /var/lib/redpanda/<bundle-name>
    ```


When you’ve finished troubleshooting, remove the diagnostics bundle from your host machine:

```bash
rm -r diagnostics-bundle
```

For a description of the files and directories, see [Contents of the diagnostics bundle](#contents-of-the-diagnostics-bundle).

## [](#inspect-the-diagnostics-bundle)Inspect the diagnostics bundle

This section provides some useful data points to check while troubleshooting.

### [](#view-the-version-of-redpanda-on-all-brokers)View the version of Redpanda on all brokers

```bash
cat admin/brokers.json | jq '.[] | .version'
```

Example output:

```json
"v23.1.1"
"v23.1.1"
"v23.1.1"
```

### [](#view-the-maintenance-status-of-all-brokers)View the maintenance status of all brokers

```bash
cat admin/brokers.json | jq '.[] | .node_id, .maintenance_status'
```

Example output

```json
0
{
  "draining": false,
  "finished": false,
  "errors": false,
  "partitions": 0,
  "eligible": 0,
  "transferring": 0,
  "failed": 0
}
1
{
  "draining": false,
  "finished": false,
  "errors": false,
  "partitions": 0,
  "eligible": 0,
  "transferring": 0,
  "failed": 0
}
2
{
  "draining": false,
  "finished": false,
  "errors": false,
  "partitions": 0,
  "eligible": 0,
  "transferring": 0,
  "failed": 0
}
```

### [](#view-the-cluster-configuration)View the cluster configuration

```bash
cat admin/cluster_config.json | jq
```

Example output

```json
{
  "abort_index_segment_size": 50000,
  "abort_timed_out_transactions_interval_ms": 10000,
  "admin_api_require_auth": false,
  "aggregate_metrics": false,
  "alter_topic_cfg_timeout_ms": 5000,
  "append_chunk_size": 16384,
  "auto_create_topics_enabled": false,
  "cloud_storage_access_key": null,
  "cloud_storage_api_endpoint": null,
  "cloud_storage_api_endpoint_port": 443,
  "cloud_storage_azure_container": null,
  "cloud_storage_azure_shared_key": null,
  "cloud_storage_azure_storage_account": null,
  "cloud_storage_bucket": null,
  ...
  "target_quota_byte_rate": 2147483648,
  "tm_sync_timeout_ms": 10000,
  "topic_fds_per_partition": 5,
  "topic_memory_per_partition": 1048576,
  "topic_partitions_per_shard": 1000,
  "topic_partitions_reserve_shard0": 2,
  "transaction_coordinator_cleanup_policy": "delete",
  "transaction_coordinator_delete_retention_ms": 604800000,
  "transaction_coordinator_log_segment_size": 1073741824,
  "transactional_id_expiration_ms": 604800000,
  "tx_log_stats_interval_s": 10,
  "tx_timeout_delay_ms": 1000,
  "wait_for_leader_timeout_ms": 5000,
  "zstd_decompress_workspace_bytes": 8388608
}
```

### [](#check-enterprise-license-keys)Check Enterprise license keys

```bash
cat admin/license.json | jq
```

Example output

```json
{
  "loaded": false,
  "license": {
    "format_version": 0,
    "org": "",
    "type": "",
    "expires": 0,
    "sha256": ""
  }
}
```

### [](#view-metadata-about-the-redpanda-data-directory)View metadata about the Redpanda data directory

To check the size of the directories and look for anomalies:

```bash
cat du.txt
```

Example output

```none
33M	/var/lib/redpanda/data/redpanda/kvstore/0_0
33M	/var/lib/redpanda/data/redpanda/kvstore
33M	/var/lib/redpanda/data/redpanda/controller/0_0
33M	/var/lib/redpanda/data/redpanda/controller
65M	/var/lib/redpanda/data/redpanda
65M	/var/lib/redpanda/data
```

To check the file permissions, file size, and last modification date of the files:

```bash
cat data-dir.txt | jq
```

Example output

```json
{
  "/var/lib/redpanda/data": {
    "size": "4.096kB",
    "mode": "dgrwxrwxrwx",
    "modified": "2023-02-02 15:21:12.430878371 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/config_cache.yaml": {
    "size": "340B",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:22.434878593 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/pid.lock": {
    "size": "2B",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:10.502878322 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.650878326 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/controller": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.650878326 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/controller/0_0": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:12.346878368 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log": {
    "size": "4.096kB",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:32.450878771 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/kvstore": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.590878324 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/kvstore/0_0": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.602878325 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log": {
    "size": "8.192kB",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:32.458878772 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/startup_log": {
    "size": "26B",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:10.510878323 +0000 UTC",
    "user": "",
    "group": "redpanda"
  }
}
```

### [](#view-cluster-metadata)View cluster metadata

```bash
cat kafka.json | jq '.[0]'
```

Example output

```json
{
  "Name": "metadata",
  "Response": {
    "Cluster": "redpanda.14a3f9b6-1c74-4ffd-806a-4ab48db78120",
    "Controller": 0,
    "Brokers": [
      {
        "NodeID": 0,
        "Port": 9093,
        "Host": "redpanda-0.redpanda.<namespace>.svc.cluster.local.",
        "Rack": null
      },
      {
        "NodeID": 1,
        "Port": 9093,
        "Host": "redpanda-1.redpanda.<namespace>.svc.cluster.local.",
        "Rack": null
      },
      {
        "NodeID": 2,
        "Port": 9093,
        "Host": "redpanda-2.redpanda.<namespace>.svc.cluster.local.",
        "Rack": null
      }
    ],
    "Topics": {}
  },
  "Error": null
}
```

### [](#view-topic-and-broker-configurations)View topic and broker configurations

```bash
cat kafka.json | jq '.[1:]'
```

Example output

```json
[
  {
    "Name": "topic_configs",
    "Response": null,
    "Error": null
  },
  {
    "Name": "broker_configs",
    "Response": [
      {
        "Name": "0",
        "Configs": [
          {
            "Key": "listeners",
            "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094",
            "Sensitive": false,
            "Source": "STATIC_BROKER_CONFIG",
            "Synonyms": [
              {
                "Key": "kafka_api",
                "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094",
                "Source": "STATIC_BROKER_CONFIG"
              },
              {
                "Key": "kafka_api",
                "Value": "plain://127.0.0.1:9092",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "advertised.listeners",
            "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092",
            "Sensitive": false,
            "Source": "STATIC_BROKER_CONFIG",
            "Synonyms": [
              {
                "Key": "advertised_kafka_api",
                "Value": "internal://redpanda-0.redpanda.<namespace>.svc.cluster.local.:9093,default://203.0.113.3:31092",
                "Source": "STATIC_BROKER_CONFIG"
              },
              {
                "Key": "advertised_kafka_api",
                "Value": "",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.segment.bytes",
            "Value": "134217728",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "log_segment_size",
                "Value": "134217728",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.retention.bytes",
            "Value": "18446744073709551615",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "retention_bytes",
                "Value": "18446744073709551615",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.retention.ms",
            "Value": "604800000",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "delete_retention_ms",
                "Value": "604800000",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "num.partitions",
            "Value": "1",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "default_topic_partitions",
                "Value": "1",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "default.replication.factor",
            "Value": "1",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "default_topic_replications",
                "Value": "1",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.dirs",
            "Value": "/var/lib/redpanda/data",
            "Sensitive": false,
            "Source": "STATIC_BROKER_CONFIG",
            "Synonyms": [
              {
                "Key": "data_directory",
                "Value": "/var/lib/redpanda/data",
                "Source": "STATIC_BROKER_CONFIG"
              }
            ]
          },
          {
            "Key": "auto.create.topics.enable",
            "Value": "false",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "auto_create_topics_enabled",
                "Value": "false",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          }
        ],
        "Err": null
      },
      {
        "Name": "1",
        "Configs": [
          ...
        ]
        ...
      },
      {
        "Name": "1",
        "Configs": [
          ...
        ]
        ...
      },
    ],
    "Error": null
  },
  {
    "Name": "log_start_offsets",
    "Response": {},
    "Error": null
  },
  {
    "Name": "last_stable_offsets",
    "Response": {},
    "Error": null
  },
  {
    "Name": "high_watermarks",
    "Response": {},
    "Error": null
  },
  {
    "Name": "groups",
    "Response": null,
    "Error": null
  }
]
```

### [](#view-the-redpanda-logs)View the Redpanda logs

```bash
cat logs/redpanda-0.txt # logs/redpanda-1.txt logs/redpanda-2.txt
```

### [](#check-for-clock-drift)Check for clock drift

```bash
cat ntp.txt | jq
```

Use the output to check for clock drift. For details about how NTP works, see the [NTP documentation](http://www.ntp.org/ntpfaq/NTP-s-algo.htm).

Example output

```json
{
  "host": "pool.ntp.org",
  "roundTripTimeMs": 3,
  "remoteTimeUTC": "2023-02-02T15:22:51.763175934Z",
  "localTimeUTC": "2023-02-02T15:22:51.698044603Z",
  "precisionMs": 0,
  "offset": -458273
}
```

### [](#view-kubernetes-manifests)View Kubernetes manifests

```bash
tree k8s
```

Example output

```none
k8s
├── configmaps.json
├── endpoints.json
├── events.json
├── limitranges.json
├── persistentvolumeclaims.json
├── pods.json
├── replicationcontrollers.json
├── resourcequotas.json
├── serviceaccounts.json
└── services.json
```

## [](#contents-of-the-diagnostics-bundle)Contents of the diagnostics bundle

The diagnostics bundle includes the following files and directories:

> 📝 **NOTE**
>
> Redpanda collects some data from the Kubernetes API. To communicate with the Kubernetes API, Redpanda requires a ClusterRole attached to the default ServiceAccount for the Pods. The files and directories that are generated only when the ClusterRole exists are labeled **Requires ClusterRole**.

| File or Directory | Description |
| --- | --- |
| /admin | Cluster and broker configurations, cluster health data, and license key information.Requires ClusterRole. |
| /controller | Binary-encoded replicated logs that contain the history of configuration changes as well as internal settings.Redpanda can replay the events that took place in the cluster to arrive at a similar state. |
| data-dir.txt | Metadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed. |
| du.txt | The disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command. |
| /k8s | Kubernetes manifests for all resources in the given Kubernetes namespace.Requires ClusterRole. |
| kafka.json | Kafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits. |
| /logs | Logs of each Pod in the given Kubernetes namespace.If --logs-since is passed, only the logs within the given timeframe are included.Requires ClusterRole. |
| /metrics | Prometheus metrics from both the /metrics endpoint and the public_metrics endpoint.One directory for each broker’s metrics.Requires ClusterRole. |
| ntp.txt | The NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed. |
| /proc | CPU details of the broker on which the rpk debug bundle command was executed.The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores. |
| redpanda.yaml | The Redpanda configuration file of the broker on which the rpk debug bundle command was executed.Sensitive data is removed and replaced with (REDACTED). |
| resource-usage.json | Redpanda resource usage data, such as CPU usage and free memory available. |

## [](#suggested-reading)Suggested reading

-   [`rpk debug bundle`](https://docs.redpanda.com/streaming/23.3/reference/rpk/rpk-debug/rpk-debug-bundle/)

-   [jq manual](https://stedolan.github.io/jq/manual/#Basicfilters)