Streaming

Enable the PVCUnbinder

The PVCUnbinder is an emergency backstop for Redpanda clusters that use PersistentVolumes (PVs) for the Redpanda data directory. When a node running a Redpanda Pod suddenly goes offline, the PVCUnbinder detects the lost node, retains the associated PV, and removes the corresponding PersistentVolumeClaim (PVC). This workflow allows the Redpanda Pod to be rescheduled on a new node without losing critical data.

The PVCUnbinder is intended only for emergency scenarios (for example, node hardware or infrastructure failures). Never use the PVCUnbinder as a routine method for removing brokers. If you want to remove brokers, see Decommission brokers for the correct procedure.

Why use the PVCUnbinder?

If a worker node hosting a Redpanda Pod suddenly fails or disappears, Kubernetes might leave the associated PV and PVC in an attached or in-use state. Without the PVCUnbinder (or manual intervention), the Redpanda Pod cannot safely reschedule to another node because the volume is still recognized as occupied. Also, the default reclaim policy might delete the volume, risking data loss. The PVCUnbinder automates the steps needed to retain the volume and remove the stale PVC, so Redpanda Pods can move to healthy nodes without losing the data in the original PV.

How the PVCUnbinder works

When the PVCUnbinder detects events that indicate a Node resource is no longer available, it does the following:

For each Redpanda Pod on that Node, it identifies the PVC (if any) the Pod was using for its storage.
It sets the reclaim policy of the affected PersistentVolume (PV) to Retain.
It deletes the associated PersistentVolumeClaim (PVC) to allow the Redpanda broker Pod to reschedule onto a new, operational node.

flowchart TB %% Define classes classDef systemAction fill:#F6FBF6,stroke:#25855a,stroke-width:2px,color:#20293c,rx:5,ry:5 A[Node fails] --> B{Is Node
running Redpanda?}:::systemAction B -- Yes --> C[Identify Redpanda Pod PVC]:::systemAction C --> D[Set PV reclaim policy to 'Retain']:::systemAction D --> E[Delete PVC]:::systemAction E --> F[Redpanda Pod
is rescheduled]:::systemAction B -- No --> G[Ignore event]:::systemAction

Enable the PVCUnbinder

Operator
Helm

In Operator-managed deployments, the PVCUnbinder runs as a controller inside the Redpanda Operator process, not as a sidecar in each Redpanda broker Pod. Enable it by passing --unbind-pvcs-after to the operator binary through the operator chart’s additionalCmdFlags.

The Redpanda custom resource (cluster.redpanda.com/v1alpha2) does not expose a clusterSpec.statefulset.sideCars.pvcUnbinder field. The Kubernetes API server silently drops the key when the CR is applied, so setting it there has no effect. Enable the PVCUnbinder on the operator deployment itself, as shown below.

--values
--set

operator-values.yaml

additionalCmdFlags:
  - --unbind-pvcs-after=60s (1)

1	`--unbind-pvcs-after`: How long a Redpanda Pod must be stuck in `Pending` with a volume node-affinity scheduling failure before the operator unbinds its PVC. A value of `0` (the default) disables the controller.

helm upgrade --install redpanda-controller redpanda/operator \
  --namespace <namespace> \
  --create-namespace \
  --set 'additionalCmdFlags={--unbind-pvcs-after=60s}' (1)

1	`--unbind-pvcs-after`: How long a Redpanda Pod must be stuck in `Pending` with a volume node-affinity scheduling failure before the operator unbinds its PVC. A value of `0` (the default) disables the controller.

Optional flags you can append to additionalCmdFlags:

--allow-pv-rebinding: After unbinding the PVC, also clear the PV’s claimRef so that the PV can be re-bound if the original node returns. Avoid when using LocalPathProvisioner or hostPath volumes.
--unbinder-label-selector=<selector>: Restrict the PVCUnbinder to Pods matching the given Kubernetes label selector.

The operator chart’s default RBAC bundle already includes the permissions the PVCUnbinder needs to read Pods and PVCs and to patch PVs, so you do not need to enable any additional RBAC toggles.

--values
--set

pvcunbinder.yaml

statefulset:
  sideCars:
    pvcUnbinder:
      enabled: true (1)
      unbindAfter: 60s (2)
rbac:
  enabled: true (3)

1	`statefulset.sideCars.pvcUnbinder.enabled`: Enables the PVCUnbinder sidecar.
2	`statefulset.sideCars.pvcUnbinder.unbindAfter`: Sets the time in seconds after which the PVCUnbinder sidecar removes the PVC after the Node resource is deleted.
3	`rbac.enabled`: Creates the required RBAC rules for the PVCUnbinder to monitor the Node resources and update PVCs and PVs.

helm upgrade --install redpanda redpanda/redpanda \
  --namespace <namespace> \
  --create-namespace \
  --set statefulset.sideCars.pvcUnbinder.enabled=true \ (1)
  --set statefulset.sideCars.pvcUnbinder.unbindAfter=60s\ (2)
  --set rbac.enabled=true (3)

1	`statefulset.sideCars.pvcUnbinder.enabled`: Enables the PVCUnbinder sidecar.
2	`statefulset.sideCars.pvcUnbinder.unbindAfter`: Sets the time in seconds after which the PVCUnbinder sidecar removes the PVC after the Node resource is deleted.
3	`rbac.enabled`: Creates the required RBAC rules for the PVCUnbinder to monitor the Node resources and update PVCs and PVs.

Test the PVCUnbinder

Test the PVCUnbinder sidecar by deleting a Node resource:
```
kubectl delete node <node-name>
```
This step is for testing purposes only.
Monitor the PVCUnbinder logs.

For a Helm-only deployment, the controller runs in the broker Pod’s sidecars container:
```
kubectl logs <pod-name> --namespace <namespace> -c sidecars
```
For an Operator-managed deployment, the controller runs in the Redpanda Operator Pod:
```
kubectl logs deployment/redpanda-controller --namespace <operator-namespace>
```
You should see that the PVCUnbinder successfully deleted the PVC of the Pod that was running on the deleted Node resource.
```
kubectl get persistentvolumeclaim --namespace <namespace>
```
Verify that the reclaim policy of the PV is set to Retain to allow you to recover the node, if necessary:
```
kubectl get persistentvolume --namespace <namespace>
```

After the PVCUnbinder has finished, decommission the broker that was removed from the node. This is necessary to prevent a potential loss of quorum and ensure cluster stability.

Make sure to use the --force flag when decommissioning the broker with rpk redpanda admin brokers decommission. This flag is required when the broker is no longer running.