Install the Nodewatcher Controller

The Nodewatcher controller is an emergency backstop for Redpanda clusters that use PersistentVolumes (PVs) for the Redpanda data directory. When a node running a Redpanda Pod suddenly goes offline, Nodewatcher detects the lost node, retains the associated PV, and removes the corresponding PersistentVolumeClaim (PVC). This workflow allows the Redpanda Pod to be rescheduled on a new node without losing critical data.

The Nodewatcher controller is intended only for emergency scenarios (for example, node hardware or infrastructure failures). Never use the Nodewatcher controller as a routine method for removing brokers. If you want to remove brokers, see Decommission brokers for the correct procedure.

Why use Nodewatcher?

If a worker node hosting a Redpanda Pod suddenly fails or disappears, Kubernetes might leave the associated PV and PVC in an attached or in-use state. Without Nodewatcher (or manual intervention), the Redpanda Pod cannot safely reschedule to another node because the volume is still recognized as occupied. Also, the default reclaim policy might delete the volume, risking data loss. Nodewatcher automates the steps needed to retain the volume and remove the stale PVC, so Redpanda Pods can move to healthy nodes without losing the data in the original PV.

How Nodewatcher works

When the controller detects events that indicate a Node resource is no longer available, it does the following:

  • For each Redpanda Pod on that Node, it identifies the PVC (if any) the Pod was using for its storage.

  • It sets the reclaim policy of the affected PersistentVolume (PV) to Retain.

  • It deletes the associated PersistentVolumeClaim (PVC) to allow the Redpanda broker Pod to reschedule onto a new, operational node.

flowchart TB %% Define classes classDef systemAction fill:#F6FBF6,stroke:#25855a,stroke-width:2px,color:#20293c,rx:5,ry:5 A[Node fails] --> B{Is Node
running Redpanda?}:::systemAction B -- Yes --> C[Identify Redpanda Pod PVC]:::systemAction C --> D[Set PV reclaim policy to 'Retain']:::systemAction D --> E[Delete PVC]:::systemAction E --> F[Redpanda Pod
is rescheduled]:::systemAction B -- No --> G[Ignore event]:::systemAction

Install Nodewatcher

  • Helm + Operator

  • Helm

You can install the Nodewatcher controller as part of the Redpanda Operator or as a sidecar on each Pod that runs a Redpanda broker. When you install the controller as part of the Redpanda Operator, the controller monitors all Redpanda clusters running in the same namespace as the Redpanda Operator. If you want the controller to manage only a single Redpanda cluster, install it as a sidecar on each Pod that runs a Redpanda broker, using the Redpanda resource.

To install the Nodewatcher controller as part of the Redpanda Operator:

  1. Deploy the Redpanda Operator with the Nodewatcher controller:

    helm repo add redpanda https://charts.redpanda.com
helm repo update
helm upgrade --install redpanda-controller redpanda/operator \
  --namespace <namespace> \
  --set image.tag=v2.3.6-24.3.3 \
  --create-namespace \
  --set additionalCmdFlags={--additional-controllers="nodeWatcher"} \
  --set rbac.createAdditionalControllerCRs=true

    • --additional-controllers="nodeWatcher": Enables the Nodewatcher controller.

    • --rbac.createAdditionalControllerCRs=true: Creates the required RBAC rules for the Redpanda Operator to monitor the Node resources and update PVCs and PVs.

  2. Deploy a Redpanda resource:

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec: {}
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

To install the Decommission controller as a sidecar:

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    statefulset:
      sideCars:
        controllers:
          enabled: true
          run:
            - "nodeWatcher"
    rbac:
      enabled: true

  • statefulset.sideCars.controllers.enabled: Enables the controllers sidecar.

  • statefulset.sideCars.controllers.run: Enables the Nodewatcher controller.

  • rbac.enabled: Creates the required RBAC rules for the controller to monitor the Node resources and update PVCs and PVs.

  • --values

  • --set

decommission-controller.yaml
statefulset:
  sideCars:
    controllers:
      enabled: true
      run:
        - "nodeWatcher"
rbac:
  enabled: true

  • statefulset.sideCars.controllers.enabled: Enables the controllers sidecar.

  • statefulset.sideCars.controllers.run: Enables the Nodewatcher controller.

  • rbac.enabled: Creates the required RBAC rules for the controller to monitor the Node resources and update PVCs and PVs.

helm upgrade --install redpanda redpanda/redpanda \
  --namespace <namespace> \
  --create-namespace \
  --set statefulset.sideCars.controllers.enabled=true \
  --set statefulset.sideCars.controllers.run={"nodeWatcher"} \
  --set rbac.enabled=true

  • statefulset.sideCars.controllers.enabled: Enables the controllers sidecar.

  • statefulset.sideCars.controllers.run: Enables the Nodewatcher controller.

  • rbac.enabled: Creates the required RBAC rules for the controller to monitor the Node resources and update PVCs and PVs.

Test the Nodewatcher controller

  1. Test the Nodewatcher controller by deleting a Node resource:

    kubectl delete node <node-name>
    This step is for testing purposes only.

  2. Monitor the logs of the Nodewatcher controller:

    • If you’re running the Nodewatcher controller as part of the Redpanda Operator:

      kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace <namespace>

    • If you’re running the Nodewatcher controller as a sidecar:

      kubectl logs <pod-name> --namespace <namespace> -c redpanda-controllers

    You should see that the controller successfully deleted the PVC of the Pod that was running on the deleted Node resource.

    kubectl get persistentvolumeclaim --namespace <namespace>

  3. Verify that the reclaim policy of the PV is set to Retain to allow you to recover the node, if necessary:

    kubectl get persistentvolume --namespace <namespace>

After the Nodewatcher controller has finished, decommission the broker that was removed from the node. This is necessary to prevent a potential loss of quorum and ensure cluster stability.

Make sure to use the --force flag when decommissioning the broker with rpk redpanda admin brokers decommission. This flag is required when the broker is no longer running.