Enable the PVCUnbinder

The PVCUnbinder is an emergency backstop for Redpanda clusters that use PersistentVolumes (PVs) for the Redpanda data directory. When a node running a Redpanda Pod suddenly goes offline, the PVCUnbinder detects the lost node, retains the associated PV, and removes the corresponding PersistentVolumeClaim (PVC). This workflow allows the Redpanda Pod to be rescheduled on a new node without losing critical data.

The PVCUnbinder is intended only for emergency scenarios (for example, node hardware or infrastructure failures). Never use the PVCUnbinder as a routine method for removing brokers. If you want to remove brokers, see Decommission brokers for the correct procedure.

Why use the PVCUnbinder?

If a worker node hosting a Redpanda Pod suddenly fails or disappears, Kubernetes might leave the associated PV and PVC in an attached or in-use state. Without the PVCUnbinder (or manual intervention), the Redpanda Pod cannot safely reschedule to another node because the volume is still recognized as occupied. Also, the default reclaim policy might delete the volume, risking data loss. The PVCUnbinder automates the steps needed to retain the volume and remove the stale PVC, so Redpanda Pods can move to healthy nodes without losing the data in the original PV.

How the PVCUnbinder works

When the PVCUnbinder detects events that indicate a Node resource is no longer available, it does the following:

  • For each Redpanda Pod on that Node, it identifies the PVC (if any) the Pod was using for its storage.

  • It sets the reclaim policy of the affected PersistentVolume (PV) to Retain.

  • It deletes the associated PersistentVolumeClaim (PVC) to allow the Redpanda broker Pod to reschedule onto a new, operational node.

flowchart TB %% Define classes classDef systemAction fill:#F6FBF6,stroke:#25855a,stroke-width:2px,color:#20293c,rx:5,ry:5 A[Node fails] --> B{Is Node
running Redpanda?}:::systemAction B -- Yes --> C[Identify Redpanda Pod PVC]:::systemAction C --> D[Set PV reclaim policy to 'Retain']:::systemAction D --> E[Delete PVC]:::systemAction E --> F[Redpanda Pod
is rescheduled]:::systemAction B -- No --> G[Ignore event]:::systemAction

Enable the PVCUnbinder

  • Operator

  • Helm

To enable the PVCUnbinder:

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    statefulset:
      sideCars:
        pvcUnbinder:
          enabled: true (1)
          unbindAfter: 60s (2)
    rbac:
      enabled: true (3)
1 statefulset.sideCars.pvcUnbinder.enabled: Enables the PVCUnbinder sidecar.
2 statefulset.sideCars.pvcUnbinder.unbindAfter: Sets the time in seconds after which the PVCUnbinder sidecar removes the PVC after the Node resource is deleted.
3 rbac.enabled: Creates the required RBAC rules for the PVCUnbinder to monitor the Node resources and update PVCs and PVs.

  • --values

  • --set

pvcunbinder.yaml
statefulset:
  sideCars:
    pvcUnbinder:
      enabled: true (1)
      unbindAfter: 60s (2)
rbac:
  enabled: true (3)
1 statefulset.sideCars.pvcUnbinder.enabled: Enables the PVCUnbinder sidecar.
2 statefulset.sideCars.pvcUnbinder.unbindAfter: Sets the time in seconds after which the PVCUnbinder sidecar removes the PVC after the Node resource is deleted.
3 rbac.enabled: Creates the required RBAC rules for the PVCUnbinder to monitor the Node resources and update PVCs and PVs. 
helm upgrade --install redpanda redpanda/redpanda \
  --namespace <namespace> \
  --create-namespace \
  --set statefulset.sideCars.pvcUnbinder.enabled=true \ (1)
  --set statefulset.sideCars.pvcUnbinder.unbindAfter=60s\ (2)
  --set rbac.enabled=true (3)
1 statefulset.sideCars.pvcUnbinder.enabled: Enables the PVCUnbinder sidecar.
2 statefulset.sideCars.pvcUnbinder.unbindAfter: Sets the time in seconds after which the PVCUnbinder sidecar removes the PVC after the Node resource is deleted.
3 rbac.enabled: Creates the required RBAC rules for the PVCUnbinder to monitor the Node resources and update PVCs and PVs.

Test the PVCUnbinder sidecar

  1. Test the PVCUnbinder sidecar by deleting a Node resource:

    kubectl delete node <node-name>
    This step is for testing purposes only.

  2. Monitor the logs of the PVCUnbinder sidecar:

    kubectl logs <pod-name> --namespace <namespace> -c sidecars

    You should see that the PVCUnbinder successfully deleted the PVC of the Pod that was running on the deleted Node resource.

    kubectl get persistentvolumeclaim --namespace <namespace>

  3. Verify that the reclaim policy of the PV is set to Retain to allow you to recover the node, if necessary:

    kubectl get persistentvolume --namespace <namespace>

After the PVCUnbinder has finished, decommission the broker that was removed from the node. This is necessary to prevent a potential loss of quorum and ensure cluster stability.

Make sure to use the --force flag when decommissioning the broker with rpk redpanda admin brokers decommission. This flag is required when the broker is no longer running.