Streaming

Monitor Cluster Load with the Load Factor Dashboard

Starting in Redpanda v26.2, the Redpanda Load Factor Grafana dashboard shows how close each broker is running to the capacity of its host. The dashboard compares live utilization against each resource’s capacity and reports the result as a percentage, the load factor, for each resource: CPU, I/O scheduler, disk read and write IOPS, memory, network bandwidth, and client connections. For disk and network, that capacity is estimated from the broker’s cloud instance type; the other resources use fixed or calculated limits (CPU utilization is already a percentage, memory reaches 100% when available memory falls to 10% of total, and connections use the lower of 1,000 per GiB of memory or 3,000 per shard). Use it to see how much headroom a cluster has, to decide when to scale, and to identify which resource saturates first as traffic grows.

After reading this page, you will be able to:

  • Identify the metrics endpoint and cluster property the Load Factor dashboard requires

  • Import the Redpanda Load Factor dashboard into Grafana

  • Interpret the load factor to decide when a cluster needs more capacity

Prerequisites

How load factor works

Traditional utilization metrics tell you how busy a resource is, but not how much capacity remains. Load factor closes that gap by comparing utilization against what the host can deliver:

  • Instance capacity metrics estimate what the broker’s host can sustain. Redpanda recognizes the cloud instance type it runs on and emits the vendor’s nominal figures for it: vCPUs, memory, disk size, read and write IOPS, and network throughput (the redpanda_instance_* metric family, labeled with cloud_provider and instance_type). Capacity figures are emitted only when Redpanda recognizes the instance type.

  • Host OS metrics report actual disk activity from the host’s /proc: read and write counts and bytes for the block devices that back the Redpanda data and cache directories (the redpanda_host_diskstats_* family, gated by the enable_host_metrics cluster property). The redpanda_io_queue_config_* metrics report the I/O rates measured for the host during startup.

These metric families are published on both metrics endpoints. On the internal endpoint (/metrics), the same metrics carry the vectorized_ prefix instead of redpanda_.

The dashboard computes a load factor for each resource and highlights the highest one, which shows the resource that saturates first. A load factor near 100% means the cluster is running close to its redline on that resource: sustained growth requires scaling up or out.

Redpanda does not emit a single load_factor metric. The load factor is computed by the dashboard from the metric families above.

Limitations

  • Capacity estimates are nominal: For disk and network, capacity estimates are the vendor’s nominal figures for the detected instance type. These figures are often achievable, but the actual capacity available to Redpanda may differ, for example, when achieved device performance differs from the vendor figures, or when another process on the same host consumes some of the resource.

  • Unrecognized instance types: When Redpanda does not recognize the instance type, capacity gauges are not emitted and the dashboard cannot compute a load factor for the affected resources.

  • Host resource detection works only on AWS: Load factors that depend on detected host resources are not reported on other cloud providers or on self-hosted hardware. You can modify the dashboard to use fixed, known values as the denominator in the load factor calculation instead.

Import the dashboard

  1. Generate the dashboard JSON with rpk (v26.2.2 or later):

    rpk generate grafana-dashboard --dashboard load-factor > redpanda-load-factor.json

    rpk downloads the dashboard from the Redpanda observability repository and verifies its checksum. To skip rpk, download the latest Redpanda-Load-Factor-Dashboard.json directly.

  2. In Grafana, go to Dashboards, click New, and select Import.

  3. Upload the JSON file, select your Prometheus data source, and click Import.

The dashboard appears as Redpanda - Load Factor (Public Metrics).

If your Prometheus scrapes the internal metrics endpoint (/metrics) rather than /public_metrics, import the internal-metrics variant instead. That variant queries the vectorized_ metric series and appears in Grafana as Redpanda - Load Factor.

Read the dashboard

The top row answers the headline question:

  • Load Factor: The current load factor per resource, as a percentage of estimated capacity.

  • Highest Load Factor: The single highest load factor across resources, which shows how close the broker is to its first bottleneck.

The information panels describe the environment the estimates are based on:

  • Host Info and Host Metrics Info: The host-level statistics feeding the disk and network factors. If this panel warns that host_metrics_info is missing, check the prerequisites: Prometheus may not be scraping the metrics endpoint the dashboard variant expects, or enable_host_metrics may be disabled.

  • Instance Info: The detected cloud provider and instance type, and whether capacity estimates are available for it.

  • IO Properties: The I/O rates measured for the host during startup.

The remaining panels break each factor down over time so you can attribute load to a resource: reactor and CPU utilization (per instance and per shard), I/O scheduler utilization, disk read and write IOPS, available memory, client connections, and network receive and transmit rates.

Next steps

Suggested labs

Search all labs