Skip to main content
Version: 23.1

Diagnostics Bundles in Kubernetes

A diagnostics bundle is a ZIP file with data that can help debug and diagnose issues with a Redpanda cluster, a broker, or the machines on which the brokers are running. You can use this file to debug issues yourself, or you can send it to the Redpanda support team to help resolve your issue.

Prerequisites​

Most files in the diagnostics bundle are JSON files. To make it easier to read these files, this guide uses jq. To install jq, see the jq downloads page.

These examples use the default values in Helm chart. If you've customized the Helm chart, you may need to provide custom values and/or flags.

Generate a diagnostics bundle​

To generate a diagnostics bundle, execute the rpk debug bundle command inside the container of a Redpanda broker.

  1. Create a ClusterRole to allow Redpanda to collect information from the Kubernetes API:

    serviceaccount.yaml
    serviceAccount:
      create: true
    rbac:
      enabled: true
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
      --values serviceaccount.yaml --reuse-values
    tip

    If you aren't using the Helm chart, you can create the ClusterRole manually:

    kubectl create clusterrolebinding redpanda --clusterrole=view --serviceaccount=redpanda:default

  2. Execute the rpk debug bundle command inside a Pod container that's running a Redpanda broker.

    In this example, the command is executed on a Pod called redpanda-0 in the redpanda namespace. You can provide your own namespace in the --namespace and -n flags.

    kubectl exec -it -n redpanda redpanda-0 -c redpanda -- rpk debug bundle \
      --admin-api-tls-truststore=<path-to-root-ca> \
      --api-urls=http://redpanda-0.redpanda.redpanda.svc.cluster.local.:9644 \
      --admin-api-tls-enabled

    If you have an upload URL from the Redpanda support team, provide it in the --upload-url flag to upload your diagnostics bundle to Redpanda.

    kubectl exec -it -n redpanda redpanda-0 -c redpanda -- rpk debug bundle \
      --upload-url <url> \
      --namespace redpanda \
      --api-urls=http://redpanda-0.redpanda.redpanda.svc.cluster.local.:9644
    Example output
    Creating bundle file...

    Debug bundle saved to "/var/lib/redpanda/1675440652-bundle.zip"

  3. On your host machine, make a directory in which to save the diagnostics bundle:

    mkdir diagnostics-bundle

  4. Copy the diagnostics bundle from the Pod to your host machine:

    Replace <bundle-name> with the name of your ZIP file.

    kubectl cp redpanda/redpanda-0:/var/lib/redpanda/<bundle-name> diagnostics-bundle/<bundle-name>

  5. Unzip the file on your host machine.

    cd diagnostics-bundle
    unzip <bundle-name>

  6. Remove the diagnostics bundle from the Redpanda container:

    kubectl exec redpanda-0 -c redpanda -n redpanda -- rm /var/lib/redpanda/<bundle-name>
important

When you've finished troubleshooting, remove the diagnostics bundle from your host machine:

rm -r diagnostics-bundle

For a description of the files and directories, see Contents of the diagnostics bundle.

Inspect the diagnostics bundle​

This section provides some useful data points to check while troubleshooting.

View the version of Redpanda on all brokers​

cat admin/brokers.json | jq '.[] | .version'
Example output
"v23.1.1"
"v23.1.1"
"v23.1.1"

View the maintenance status of all brokers​

cat admin/brokers.json | jq '.[] | .node_id, .maintenance_status'
Example output
0
{
  "draining": false,
  "finished": false,
  "errors": false,
  "partitions": 0,
  "eligible": 0,
  "transferring": 0,
  "failed": 0
}
1
{
  "draining": false,
  "finished": false,
  "errors": false,
  "partitions": 0,
  "eligible": 0,
  "transferring": 0,
  "failed": 0
}
2
{
  "draining": false,
  "finished": false,
  "errors": false,
  "partitions": 0,
  "eligible": 0,
  "transferring": 0,
  "failed": 0
}

View the cluster configuration​

cat admin/cluster_config.json | jq
Example output
{
  "abort_index_segment_size": 50000,
  "abort_timed_out_transactions_interval_ms": 10000,
  "admin_api_require_auth": false,
  "aggregate_metrics": false,
  "alter_topic_cfg_timeout_ms": 5000,
  "append_chunk_size": 16384,
  "auto_create_topics_enabled": false,
  "cloud_storage_access_key": null,
  "cloud_storage_api_endpoint": null,
  "cloud_storage_api_endpoint_port": 443,
  "cloud_storage_azure_container": null,
  "cloud_storage_azure_shared_key": null,
  "cloud_storage_azure_storage_account": null,
  "cloud_storage_bucket": null,
  ...
  "target_quota_byte_rate": 2147483648,
  "tm_sync_timeout_ms": 10000,
  "topic_fds_per_partition": 5,
  "topic_memory_per_partition": 1048576,
  "topic_partitions_per_shard": 1000,
  "topic_partitions_reserve_shard0": 2,
  "transaction_coordinator_cleanup_policy": "delete",
  "transaction_coordinator_delete_retention_ms": 604800000,
  "transaction_coordinator_log_segment_size": 1073741824,
  "transactional_id_expiration_ms": 604800000,
  "tx_log_stats_interval_s": 10,
  "tx_timeout_delay_ms": 1000,
  "wait_for_leader_timeout_ms": 5000,
  "zstd_decompress_workspace_bytes": 8388608
}

Check Enterprise license keys​

cat admin/license.json | jq
Example output
{
  "loaded": false,
  "license": {
    "format_version": 0,
    "org": "",
    "type": "",
    "expires": 0,
    "sha256": ""
  }
}

View metadata about the Redpanda data directory​

To check the size of the directories and look for anomalies:

cat du.txt
Example output
33M /var/lib/redpanda/data/redpanda/kvstore/0_0
33M /var/lib/redpanda/data/redpanda/kvstore
33M /var/lib/redpanda/data/redpanda/controller/0_0
33M /var/lib/redpanda/data/redpanda/controller
65M /var/lib/redpanda/data/redpanda
65M /var/lib/redpanda/data

To check the file permissions, file size, and last modification date of the files:

cat data-dir.txt | jq
Example output
{
  "/var/lib/redpanda/data": {
    "size": "4.096kB",
    "mode": "dgrwxrwxrwx",
    "modified": "2023-02-02 15:21:12.430878371 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/config_cache.yaml": {
    "size": "340B",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:22.434878593 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/pid.lock": {
    "size": "2B",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:10.502878322 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.650878326 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/controller": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.650878326 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/controller/0_0": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:12.346878368 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log": {
    "size": "4.096kB",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:32.450878771 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/kvstore": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.590878324 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/kvstore/0_0": {
    "size": "4.096kB",
    "mode": "dgrwxr-xr-x",
    "modified": "2023-02-02 15:21:10.602878325 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log": {
    "size": "8.192kB",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:32.458878772 +0000 UTC",
    "user": "",
    "group": "redpanda"
  },
  "/var/lib/redpanda/data/startup_log": {
    "size": "26B",
    "mode": "-rw-r--r--",
    "modified": "2023-02-02 15:21:10.510878323 +0000 UTC",
    "user": "",
    "group": "redpanda"
  }
}

View cluster metadata​

cat kafka.json | jq '.[0]'
Example output
{
  "Name": "metadata",
  "Response": {
    "Cluster": "redpanda.14a3f9b6-1c74-4ffd-806a-4ab48db78120",
    "Controller": 0,
    "Brokers": [
      {
        "NodeID": 0,
        "Port": 9093,
        "Host": "redpanda-0.redpanda.redpanda.svc.cluster.local.",
        "Rack": null
      },
      {
        "NodeID": 1,
        "Port": 9093,
        "Host": "redpanda-1.redpanda.redpanda.svc.cluster.local.",
        "Rack": null
      },
      {
        "NodeID": 2,
        "Port": 9093,
        "Host": "redpanda-2.redpanda.redpanda.svc.cluster.local.",
        "Rack": null
      }
    ],
    "Topics": {}
  },
  "Error": null
}

View topic and broker configurations​

cat kafka.json | jq '.[1:]'
Example output
[
  {
    "Name": "topic_configs",
    "Response": null,
    "Error": null
  },
  {
    "Name": "broker_configs",
    "Response": [
      {
        "Name": "0",
        "Configs": [
          {
            "Key": "listeners",
            "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094",
            "Sensitive": false,
            "Source": "STATIC_BROKER_CONFIG",
            "Synonyms": [
              {
                "Key": "kafka_api",
                "Value": "internal://0.0.0.0:9093,default://0.0.0.0:9094",
                "Source": "STATIC_BROKER_CONFIG"
              },
              {
                "Key": "kafka_api",
                "Value": "plain://127.0.0.1:9092",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "advertised.listeners",
            "Value": "internal://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093,default://203.0.113.3:31092",
            "Sensitive": false,
            "Source": "STATIC_BROKER_CONFIG",
            "Synonyms": [
              {
                "Key": "advertised_kafka_api",
                "Value": "internal://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093,default://203.0.113.3:31092",
                "Source": "STATIC_BROKER_CONFIG"
              },
              {
                "Key": "advertised_kafka_api",
                "Value": "",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.segment.bytes",
            "Value": "134217728",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "log_segment_size",
                "Value": "134217728",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.retention.bytes",
            "Value": "18446744073709551615",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "retention_bytes",
                "Value": "18446744073709551615",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.retention.ms",
            "Value": "604800000",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "delete_retention_ms",
                "Value": "604800000",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "num.partitions",
            "Value": "1",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "default_topic_partitions",
                "Value": "1",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "default.replication.factor",
            "Value": "1",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "default_topic_replications",
                "Value": "1",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          },
          {
            "Key": "log.dirs",
            "Value": "/var/lib/redpanda/data",
            "Sensitive": false,
            "Source": "STATIC_BROKER_CONFIG",
            "Synonyms": [
              {
                "Key": "data_directory",
                "Value": "/var/lib/redpanda/data",
                "Source": "STATIC_BROKER_CONFIG"
              }
            ]
          },
          {
            "Key": "auto.create.topics.enable",
            "Value": "false",
            "Sensitive": false,
            "Source": "DEFAULT_CONFIG",
            "Synonyms": [
              {
                "Key": "auto_create_topics_enabled",
                "Value": "false",
                "Source": "DEFAULT_CONFIG"
              }
            ]
          }
        ],
        "Err": null
      },
      {
        "Name": "1",
        "Configs": [
          ...
        ]
        ...
      },
      {
        "Name": "1",
        "Configs": [
          ...
        ]
        ...
      },
    ],
    "Error": null
  },
  {
    "Name": "log_start_offsets",
    "Response": {},
    "Error": null
  },
  {
    "Name": "last_stable_offsets",
    "Response": {},
    "Error": null
  },
  {
    "Name": "high_watermarks",
    "Response": {},
    "Error": null
  },
  {
    "Name": "groups",
    "Response": null,
    "Error": null
  }
]

View the Redpanda logs​

cat logs/redpanda-0.txt # logs/redpanda-1.txt logs/redpanda-2.txt

Check for clock drift​

cat ntp.txt | jq

Use the output to check for clock drift. For details about how NTP works, see the NTP documentation.

Example output
{
  "host": "pool.ntp.org",
  "roundTripTimeMs": 3,
  "remoteTimeUTC": "2023-02-02T15:22:51.763175934Z",
  "localTimeUTC": "2023-02-02T15:22:51.698044603Z",
  "precisionMs": 0,
  "offset": -458273
}

View Kubernetes manifests​

tree k8s
Example output
k8s
├── configmaps.json
├── endpoints.json
├── events.json
├── limitranges.json
├── persistentvolumeclaims.json
├── pods.json
├── replicationcontrollers.json
├── resourcequotas.json
├── serviceaccounts.json
└── services.json

Contents of the diagnostics bundle​

The diagnostics bundle includes the following files and directories:

note

Redpanda collects some data from the Kubernetes API. To communicate with the Kubernetes API, Redpanda requires a ClusterRole attached to the default ServiceAccount for the Pods. The files and directories that are generated only when the ClusterRole exists are labeled Requires ClusterRole.

File or DirectoryDescription
/adminCluster and broker configurations, cluster health data, and license key information.
Requires ClusterRole.
/controllerBinary-encoded replicated logs that contain the history of configuration changes as well as internal settings.
Redpanda can replay the events that took place in the cluster to arrive at a similar state.
data-dir.txtMetadata for the Redpanda data directory of the broker on which the rpk debug bundle command was executed.
du.txtThe disk usage of the data directory of the broker on which the rpk debug bundle command was executed, as output by the du command.
/k8sKubernetes manifests for all resources in the given Kubernetes namespace.
Requires ClusterRole.
kafka.jsonKafka metadata, such as broker configuration, topic configuration, offsets, groups, and group commits.
/logsLogs of each Pod in the given Kubernetes namespace.
If --logs-since is passed, only the logs within the given timeframe are included.
Requires ClusterRole.
/metricsPrometheus metrics from both the /metrics endpoint and the public_metrics endpoint.
One directory for each broker's metrics.
Requires ClusterRole.
ntp.txtThe NTP clock delta (using ntppool as a reference) and round trip time of the broker on which the rpk debug bundle command was executed.
/procCPU details of the broker on which the rpk debug bundle command was executed.
The directory includes a cpuinfo file with CPU information such as processor model, core count, cache size, frequency, as well as an interrupts file that contains IRQ distribution across CPU cores.
redpanda.yamlThe Redpanda configuration file of the broker on which the rpk debug bundle command was executed.
Sensitive data is removed and replaced with (REDACTED).
resource-usage.jsonRedpanda resource usage data, such as CPU usage and free memory available.

Suggested reading​

What do you like about this doc?




Optional: Share your email address if we can contact you about your feedback.

Let us know what we do well:
Ask in the community
Make a contribution