Migrate from a StatefulSet to Broker Resources

By default, the Redpanda Operator manages a cluster’s broker Pods through a StatefulSet. You can migrate a running cluster so that the Redpanda Operator manages each broker through its own Broker custom resource instead. The migration happens in place and doesn’t restart brokers, so clients stay connected while it runs.

After reading this page, you will be able to:

  • Enable Broker resources on the Redpanda Operator

  • Migrate a running cluster from a StatefulSet to Broker resources without restarting brokers

  • Roll back a cluster to StatefulSet management

Broker resources are a beta feature. You must enable experimental CRDs and the --enable-broker operator flag to use them. Before you migrate a production cluster, review the limitations.

How Broker resources work

A Broker is a custom resource (API version cluster.redpanda.com/v1alpha2, kind Broker, short name brk) that represents one Redpanda broker and owns that broker’s Pod and PersistentVolumeClaims (PVCs). When a Redpanda cluster uses Broker resources:

  • The Redpanda Operator creates one Broker resource for each broker in the cluster. The Redpanda resource owns the Broker resources, and you don’t create or delete them yourself.

  • Pod and PVC names stay the same as in StatefulSet mode, for example redpanda-0 and datadir-redpanda-0.

  • You still set the number of brokers with statefulset.replicas in the Redpanda resource. The Redpanda Operator adds or removes Broker resources to match.

  • When a change requires broker restarts, the Redpanda Operator restarts one broker at a time and waits for the cluster to be healthy before it restarts the next one.

Because each broker is linked directly to its Pod and PVCs, you can decommission or replace any broker, not only the one with the highest Pod ordinal.

If the cluster uses NodePool resources, the Redpanda Operator migrates the brokers of each NodePool too. For those brokers, the NodePool’s spec.replicas sets the number of brokers instead of statefulset.replicas, the OWNER KIND column of kubectl get brokers shows NodePool, and Pod names include the pool name, for example redpanda-pool1-0. A NodePool can’t be named default, because the cluster’s own pool uses that name when it uses Broker resources.

What changes after you migrate

Some cluster operations behave differently when the cluster uses Broker resources. Review these differences before you migrate.

Operation StatefulSet Broker resources

Inspect brokers

kubectl get statefulset

kubectl get brokers. No StatefulSet exists, so kubectl rollout commands don’t apply. See Perform a Rolling Restart of Redpanda in Kubernetes.

Scale in

By default, the PVC of the removed broker is kept.

The broker is decommissioned and its PVC is deleted, together with the data on it. See Scale in.

Decommission a specific broker

Only the broker with the highest Pod ordinal can be removed.

Any broker can be decommissioned. See Decommission a broker that uses Broker resources.

Delete the Redpanda resource

By default, PVCs are kept.

PVCs are deleted unless you set the orphan deletion policy. See Keep PVCs when you delete a cluster.

Inspect the PodDisruptionBudget

kubectl get pdb shows the maxUnavailable value from the Redpanda resource.

kubectl get pdb shows minAvailable instead: the number of brokers minus the maxUnavailable value from the Redpanda resource. With the default maxUnavailable of 1, a node drain can evict one broker at a time, as with a StatefulSet.

Downgrade the Redpanda Operator

No extra steps.

Roll back to StatefulSet management first. See Roll back to StatefulSet management.

Keep PVCs when you delete a cluster

When you delete a Redpanda resource that uses Broker resources, the Redpanda Operator deletes the PVCs of all brokers. This is the default cascade deletion policy. To keep a broker’s PVC, for example to recover its data later, set the operator.redpanda.com/broker-deletion-policy annotation to orphan on its Broker resource before you delete the Redpanda resource.

To keep the PVC of one broker:

kubectl annotate broker <broker-name> --namespace <namespace> \
  operator.redpanda.com/broker-deletion-policy=orphan

To keep the PVCs of all brokers in a cluster:

kubectl annotate brokers --namespace <namespace> \
  --selector cluster.redpanda.com/cluster-name=<cluster-name> \
  operator.redpanda.com/broker-deletion-policy=orphan

Limitations

Clusters that use Broker resources have the following limitations:

  • You can’t keep a broker’s PVC when you scale in or decommission a broker. The Redpanda Operator always deletes the PVC of a decommissioned broker. The orphan deletion policy applies only when you delete the Redpanda resource.

  • You can’t trigger a rolling restart of the whole cluster. The Redpanda Operator restarts brokers one at a time only when a configuration change requires it. You can restart a single broker by deleting its Pod. See Perform a Rolling Restart of Redpanda in Kubernetes.

  • You can’t resize PVCs. The procedure in Expand PersistentVolumes depends on a StatefulSet.

If you need any of these operations, roll back to StatefulSet management first.

Prerequisites

  • Redpanda Operator v26.3.1 or later, deployed with Helm. See Deploy Redpanda for Production in Kubernetes.

  • A healthy cluster. Run the following command and confirm that the output shows Healthy: true:

    kubectl exec -n <namespace> redpanda-0 -c redpanda -- rpk cluster health
  • No rolling restart, scale operation, or configuration change in progress. The Redpanda Operator waits to migrate until every broker runs the current configuration. See The migration stays Blocked.

  • The StatefulSet has the same name as the Redpanda resource. The Redpanda Operator can’t migrate a cluster whose Redpanda resource sets fullnameOverride to a different name. See The migration fails with a StatefulSet name error.

  • If the cluster uses NodePool resources, no NodePool is named default. See [node-pools].

Enable Broker resources

The Broker CRD is installed with the Redpanda Operator’s experimental CRDs, and the Broker controller runs only when you pass the --enable-broker flag to the Redpanda Operator.

Install the Broker CRD before or together with the --enable-broker flag. If the flag is set and the CRD isn’t installed, the Redpanda Operator Pod fails to start with the error no matches for kind "Broker" in version "cluster.redpanda.com/v1alpha2".
  1. Add the following to your Redpanda Operator Helm values:

    operator-values.yaml
    crds:
      enabled: true
      experimental: true (1)
    additionalCmdFlags:
      - --enable-broker (2)
    1 Installs the Broker CRD, together with the other experimental CRDs.
    2 Starts the Broker controller. If your values already set additionalCmdFlags, add --enable-broker to the existing list instead of replacing it.
  2. Upgrade the Redpanda Operator with these values:

    helm upgrade --install redpanda-controller redpanda/operator \
      --namespace <namespace> --create-namespace \
      --version <operator-version> \
      --values operator-values.yaml

    Set <operator-version> to the version of the Redpanda Operator that you already run. If this upgrade also changes the Redpanda Operator version, the brokers restart one at a time. See Broker restarts during operator upgrades.

  3. Verify that the Broker CRD is installed:

    kubectl get crd brokers.cluster.redpanda.com

Enabling Broker resources doesn’t change existing clusters. Each cluster keeps its StatefulSet until you migrate it.

Migrate the cluster

To migrate a cluster, set the operator.redpanda.com/use-broker-cr annotation on its Redpanda resource. The Redpanda Operator then creates a Broker resource for each running Pod, transfers ownership of each Pod and its PVC to the Broker resource, and removes the StatefulSet. Pods aren’t recreated, and brokers aren’t restarted.

  1. Annotate the Redpanda resource:

    kubectl annotate redpanda <cluster-name> --namespace <namespace> \
      operator.redpanda.com/use-broker-cr=true
  2. Wait for the migration to complete:

    kubectl wait redpanda <cluster-name> --namespace <namespace> \
      --for=jsonpath='{.status.conditions[?(@.type=="BrokerMigration")].reason}'=Complete \
      --timeout=10m

    The Redpanda Operator reports progress in the BrokerMigration condition of the Redpanda resource. When the migration is complete, the condition has the reason Complete and the message StatefulSet removed; Broker CRs manage all pods. If the command times out, check the conditions of the Redpanda resource, and see Troubleshooting:

    kubectl get redpanda <cluster-name> --namespace <namespace> \
      --output jsonpath='{range .status.conditions[*]}{.type}: {.reason}: {.message}{"\n"}{end}'
  3. Verify that each broker has a Broker resource in the Running phase:

    kubectl get brokers --namespace <namespace>

    Broker resource names have a random suffix, and the BROKERID column shows the Redpanda broker ID.

    Example output
    NAME             PHASE     BROKERID   AGE   OWNER KIND   OWNER      READY
    redpanda-cnh5w   Running   1          21s   Redpanda     redpanda   True
    redpanda-r5zz9   Running   2          21s   Redpanda     redpanda   True
    redpanda-tcq7h   Running   0          21s   Redpanda     redpanda   True
  4. Verify that the StatefulSet no longer exists:

    kubectl get statefulset --namespace <namespace>
    Example output
    No resources found in <namespace> namespace.
Keep the annotation on the Redpanda resource for as long as you use Broker resources. Removing it rolls the cluster back to StatefulSet management.
A new cluster can use Broker resources from the start. Set the operator.redpanda.com/use-broker-cr: "true" annotation in the metadata.annotations of the Redpanda resource before you apply it, and the Redpanda Operator never creates a StatefulSet for it.

Roll back to StatefulSet management

To roll back, remove the operator.redpanda.com/use-broker-cr annotation. The Redpanda Operator recreates the StatefulSet, transfers ownership of the Pods and PVCs back to it, and deletes the Broker resources. The handover itself doesn’t recreate Pods, but the Redpanda Operator may restart brokers afterward. See the note at the end of this section.

Until the StatefulSet owns every broker Pod again, the PodDisruptionBudget allows no evictions, so a node drain that starts during the rollback waits until the rollback completes.

Roll back before you remove the --enable-broker flag or downgrade the Redpanda Operator to a version that doesn’t support Broker resources.

  1. Remove the annotation:

    kubectl annotate redpanda <cluster-name> --namespace <namespace> \
      operator.redpanda.com/use-broker-cr-
  2. Wait for the rollback to complete:

    kubectl wait redpanda <cluster-name> --namespace <namespace> \
      --for=jsonpath='{.status.conditions[?(@.type=="BrokerMigration")].reason}'=RolledBack \
      --timeout=10m
  3. Verify that the StatefulSet exists and that no Broker resources remain:

    kubectl get statefulset,brokers --namespace <namespace>
If you changed the cluster configuration while it used Broker resources, the restored StatefulSet doesn’t match the running Pods. After the rollback, the Redpanda Operator restarts the brokers one at a time to apply the current configuration.

Troubleshooting

Use the conditions on the Redpanda resource and on each Broker resource to diagnose problems.

The Redpanda Operator stops managing brokers

If you set the operator.redpanda.com/use-broker-cr annotation while the Redpanda Operator runs without the --enable-broker flag, the Redpanda Operator doesn’t migrate the cluster. Instead, it stops managing the brokers and reports the following message in the ResourcesSynced condition of the Redpanda resource:

Condition message
cluster is managed via Broker CRs (operator.redpanda.com/use-broker-cr annotation) but the operator is running without --enable-broker; pool management suspended — restore the flag, or roll back to StatefulSet mode before downgrading

The brokers keep running, but the Redpanda Operator doesn’t apply changes to them. To resolve this, either enable Broker resources on the Redpanda Operator, or remove the annotation.

The migration stays Blocked

The Redpanda Operator migrates a cluster only when the cluster is stable. Until then, the BrokerMigration condition has the reason Blocked, and its message names what the Redpanda Operator waits for:

  • StatefulSet rollout has not completed

  • config change pending, StatefulSet not yet updated

  • pod <pod-name> is not running the desired configuration

  • a scale or pod replacement is in progress

  • pods are pending a rolling update

Wait for the rolling restart, scale operation, or configuration change to finish. The Redpanda Operator then starts the migration without further action.

The migration fails with a StatefulSet name error

If the Redpanda resource sets fullnameOverride, the StatefulSet name differs from the Redpanda resource name, and the Redpanda Operator can’t migrate the cluster. The ResourcesSynced condition of the Redpanda resource reports an error that contains the following message:

Condition message
rendered StatefulSet name "<statefulset-name>" differs from the Redpanda name "<cluster-name>" (fullnameOverride?); Broker CR mode requires matching names

To resolve this, remove the operator.redpanda.com/use-broker-cr annotation. The cluster stays managed by its StatefulSet.

A Broker resource is in the Stuck phase

A Broker resource moves to the Stuck phase when its Pod can’t start, for example because the Pod can’t be scheduled or its container is in a crash loop. To find the cause, check the conditions of the Broker resource:

kubectl get broker <broker-name> --namespace <namespace> \
  --output jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'

For example, a Pod that can’t be scheduled reports the scheduler’s message in the PodScheduled condition, such as didn’t match pod anti-affinity rules.