Streaming

Configure Debug Bundle Management in Kubernetes

Configure debug bundle retention for Redpanda brokers in Kubernetes, and the capture options for the multicluster operator debug bundle.

Which settings apply depends on how Redpanda is deployed:

Deployment Debug bundle What you configure

Single Kubernetes cluster (Redpanda Operator or Helm chart)

Redpanda debug bundle, collected from the brokers

config.cluster.debug_bundle_auto_removal_seconds on the Redpanda resource or in Helm values. See Single Kubernetes cluster.

Stretch Cluster across several Kubernetes clusters

Both: the Redpanda debug bundle from the brokers, and the multicluster operator debug bundle from the operators

The same config.cluster.debug_bundle_auto_removal_seconds for the broker bundle, set on the StretchCluster resource. Nothing on the cluster for the operator bundle: it is written to your machine and controlled by rpk k8s multicluster bundle flags. See Stretch Cluster.

Single Kubernetes cluster

In a single-cluster deployment, the only debug bundle is the one collected from the Redpanda brokers. Set its retention and storage properties on the Redpanda resource or in Helm values.

These configuration options apply only when you trigger a debug bundle remotely. For example using Redpanda Console, rpk debug remote-bundle, or the Admin API. They do not apply to the rpk debug bundle command.

Automatically remove debug bundles

To avoid manually deleting debug bundles, you can configure the debug_bundle_auto_removal_seconds property. This cluster configuration property automatically deletes debug bundles after the specified number of seconds. By default, this property is not set, meaning debug bundles are retained indefinitely.

Only one debug bundle can exist at a time. If you generate a new debug bundle, any existing bundle from a previous run will be automatically deleted.

Changes to this property take effect immediately and do not require a cluster restart.

To set this property, use the config.cluster.debug_bundle_auto_removal_seconds field:

  • Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    config:
      cluster:
        debug_bundle_auto_removal_seconds: <seconds>

For example, to retain debug bundles for 1 day:

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    config:
      cluster:
        debug_bundle_auto_removal_seconds: 86400

Apply the changes with:

kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

Update the values.yaml file or use the --set flag to specify the property:

  • --values

  • --set

cloud-storage.yaml
config:
  cluster:
    debug_bundle_auto_removal_seconds: <seconds>

For example, to retain debug bundles for 1 day:

cloud-storage.yaml
config:
  cluster:
    debug_bundle_auto_removal_seconds: 86400

Apply the changes with:

helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values cloud-storage.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set config.cluster.debug_bundle_auto_removal_seconds=<seconds>

For example, to retain debug bundles for 1 day:

helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set config.cluster.debug_bundle_auto_removal_seconds=86400

Choose where the debug bundle is saved

The debug_bundle_storage_dir property allows you to specify a custom directory for storing debug bundles on the broker. By default, debug bundles are stored in the Redpanda data directory. Configuring a custom storage directory can help manage storage capacity and isolate debug data from operational data.

Changes to this property take effect immediately and do not require a cluster restart.

Before you change this property:

  • Ensure that your chosen directory has sufficient storage capacity to handle debug bundles.

    Factors such as the volume of logs can increase the bundle size. While it is difficult to define an exact storage requirement due to variability in bundle size, 200 MB should be sufficient for most cases.

  • Verify the directory’s permissions to ensure Redpanda can write to it. By default, Redpanda operates as the redpanda user within the redpanda group.

To set this property, use the config.cluster.debug_bundle_storage_dir field:

  • Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    config:
      cluster:
        debug_bundle_storage_dir: <path-to-directory>

For example:

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    config:
      cluster:
        debug_bundle_storage_dir: /var/log/redpanda/debug_bundles

Apply the changes with:

kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

Update the values.yaml file or use the --set flag to specify the property:

  • --values

  • --set

config:
  cluster:
    debug_bundle_storage_dir: <path-to-directory>

For example, to store debug bundles in /var/log/redpanda/debug_bundles:

config:
  cluster:
    debug_bundle_storage_dir: /var/log/redpanda/debug_bundles

Apply the changes with:

helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values values.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set config.cluster.debug_bundle_storage_dir=<path-to-directory>

For example:

helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set config.cluster.debug_bundle_storage_dir=/var/log/redpanda/debug_bundles

Stretch Cluster

A Stretch Cluster has two debug bundles: the Redpanda debug bundle from the brokers, which you configure on the StretchCluster resource, and the multicluster operator debug bundle, which has no cluster-side configuration.

Redpanda debug bundle

The same cluster properties as for a single Kubernetes cluster apply, including debug_bundle_auto_removal_seconds and debug_bundle_storage_dir. Set them under spec.config.cluster on the StretchCluster resource, which has no clusterSpec level. For example, to retain debug bundles for 1 day:

stretch-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: StretchCluster
metadata:
  name: redpanda
spec:
  config:
    cluster:
      debug_bundle_auto_removal_seconds: 86400

Apply the same StretchCluster manifest to every Kubernetes cluster, as described in Deploy a Stretch Cluster on Kubernetes.

Multicluster operator debug bundle

The rpk k8s multicluster bundle command collects the operator Pod and Deployment, logs, /metrics samples, Raft status, and TLS material from every Kubernetes cluster in a Stretch Cluster, and writes one ZIP file to your local machine. The Redpanda brokers are not involved, so the debug_bundle_auto_removal_seconds property and the storage settings above don’t apply. Delete the ZIP file yourself when you no longer need it.

Prerequisites

  • Redpanda Operator version 26.2.1 or later on every cluster.

  • rpk version 26.2 or later with the k8s plugin. rpk installs the plugin the first time you run an rpk k8s command, or you can install it in advance with rpk k8s install.

  • A kubeconfig for at least one of the Stretch Cluster’s Kubernetes clusters. The command discovers the other clusters from the kubeconfig cache Secrets that the operator keeps in its namespace.

Capture options

Set these flags when you run the command. There is no persistent configuration.

Option Default Description

--include-private-keys

off

Keep tls.key and the cached peer kubeconfig.yaml payloads in the serialized Secrets. By default both are redacted. Leave this off unless Redpanda Support asks for the keys.

--logs-size-limit, --logs-tail-lines

5M, 5000

Cap the operator log collected from each cluster, by bytes and by lines. Set --logs-size-limit to 0 or --logs-tail-lines to -1 to remove that cap. Use --skip-logs to collect no logs.

--metrics-samples, --metrics-interval

2, 10s

Number of /metrics scrapes per cluster, minimum 2, and the interval between them. Two or more samples let Redpanda Support compute counter rates, such as Raft leader changes, without a live Prometheus. Use --skip-metrics to collect no metrics.

--context

none

Repeat the flag to name the clusters to bundle. You must pass --context, --kubeconfig, or both. If you pass only --kubeconfig, every context in that file is used. Discovery runs only when exactly one context is selected. With more than one, the command skips discovery and bundles exactly those clusters, which is useful when the cached peer kubeconfigs aren’t reachable from your machine.

--namespace

redpanda

The namespace that the Redpanda Operator runs in.

-o, --output

./operator-bundle-<timestamp>.zip

Where to write the ZIP file.

For the full flag list, see the command reference. To generate a bundle, see Generate a multicluster operator debug bundle.

Next steps