Generate a Debug Bundle with rpk in Kubernetes

Use rpk to generate a debug bundle on a Redpanda cluster running in Kubernetes.

To generate a debug bundle with rpk, you have two options depending on your needs:

  • Use rpk debug bundle: Run this command directly on each broker in the cluster. This method requires access to the nodes your brokers are running on.

  • Use rpk debug remote-bundle: Run this command from a remote machine to collect diagnostics data from all brokers in the cluster. This method is ideal when you want to gather data without logging into each node individually.

Prerequisites

You must have rpk installed on your host machine.

Use rpk debug bundle

To generate a debug bundle with rpk, you can run the rpk debug bundle command on each broker in the cluster.

  1. Create a ClusterRole to allow Redpanda to collect information from the Kubernetes API:

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    serviceAccount:
      create: true
    rbac:
      enabled: true
    yaml
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
    bash
    You must deploy the Redpanda Operator with the --set rbac.createRPKBundleCRs=true flag to give it the required ClusterRoles.

    If you aren’t using the Helm chart, you can create the ClusterRole manually:

    kubectl create clusterrolebinding redpanda --clusterrole=view --serviceaccount=redpanda:default
    bash

  2. Execute the rpk debug bundle command on a broker:

    kubectl exec -it --namespace <namespace> redpanda-0 -c redpanda -- rpk debug bundle --namespace <namespace>
    bash

    If you have an upload URL from the Redpanda support team, provide it in the --upload-url flag to upload your debug bundle to Redpanda.

    kubectl exec -it --namespace <namespace> redpanda-0 -c redpanda -- rpk debug bundle \
  --upload-url <url> \
  --namespace <namespace>
    bash

    Example output:

    Creating bundle file...

Debug bundle saved to "/var/lib/redpanda/1675440652-bundle.zip"

  3. On your host machine, make a directory in which to save the debug bundle:

    mkdir debug-bundle
    bash

  4. Copy the debug bundle ZIP file to the debug-bundle directory on your host machine.

    Replace <bundle-name> with the name of your ZIP file.

    kubectl cp <namespace>/redpanda-0:/var/lib/redpanda/<bundle-name> debug-bundle/<bundle-name>.zip
    bash

  5. Unzip the file on your host machine and use it to debug your cluster.

    cd debug-bundle
unzip <bundle-name>.zip
    bash

    For guidance on reading the debug bundle, see Inspect a Debug Bundle.

  6. Remove the debug bundle from the Redpanda broker:

    kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rm /var/lib/redpanda/<bundle-name>.zip
    bash

When you’ve finished troubleshooting, remove the debug bundle from your host machine:

rm -r debug-bundle
bash

Use rpk debug remote-bundle

The rpk debug remote-bundle command allows you to remotely generate a consolidated debug bundle from all brokers in your cluster. This command is useful when you do not want to log into each broker’s node individually using rpk debug bundle.

  1. Create an rpk profile to connect to your cluster. Include all brokers you want to collect data from in the admin.hosts configuration.

    rpk profile create <profile-name> --set admin.hosts=<brokers>
    bash

  2. Check the configured addresses in your profile:

    rpk profile print
    bash

    Example output:

    profile.yml
    admin:
  hosts:
    - broker1:9644
    - broker2:9644
    - broker3:9644
    yaml

  3. Start the debug bundle process:

    rpk debug remote-bundle start --namespace <namespace>
    bash

    Replace <namespace> with the Kubernetes namespace in which your Redpanda cluster is running.

    To skip the confirmation steps, use the --no-confirm flag.

    To generate a bundle for a subset of brokers, use the -X admin.hosts flag. For example, rpk debug remote-bundle start -X admin.hosts target-broker:9644.

  4. Check the status:

    rpk debug remote-bundle status
    bash

    Example output:

    BROKER           STATUS   JOB-ID
localhost:29644  running  7f93fd6e-fc5e-46a5-8842-717542f89e59
localhost:19644  running  7f93fd6e-fc5e-46a5-8842-717542f89e59

    When the status changes to success, the process is complete.

    To cancel a debug bundle process while it’s running, use rpk debug remote-bundle cancel.

  5. When the process is complete, download the debug bundle:

    rpk debug remote-bundle download
    bash

    By default the compressed file is downloaded to your current working directory. To choose a different location or filename, use the --output flag. For example:

    rpk debug remote-bundle download --output ~/redpanda/debug-bundles/cluster1
    bash

    This command results in cluster1.zip downloaded to the ~/redpanda/debug-bundles/ directory.

Unzip the file and use the contents to debug your cluster. For guidance on reading the debug bundle, see Inspect a Debug Bundle.

Configure the debug_bundle_auto_removal_seconds property to automatically remove debug bundles after a period of time. See Automatically remove debug bundles.

Next steps

Inspect a Debug Bundle.

Suggested reading