Cloud

Monitor Cluster Load with the Load Factor Dashboard

The Redpanda Load Factor Grafana dashboard shows how close each broker in a Bring Your Own Cloud (BYOC) cluster is running to the capacity of its host. It tracks CPU, I/O scheduler, disk read and write IOPS, memory, network bandwidth, and client connections. For each resource, the dashboard compares live utilization with the resource’s capacity and reports the result as a percentage, called the load factor. Use it to see how much headroom a cluster has and which resource saturates first, so you know where to look before you scale.

A high load factor is a signal to investigate, not an automatic instruction to scale. For example, a single hot partition can push one broker close to 100% while the rest of the cluster has headroom, and adding brokers does not fix that.

After reading this page, you will be able to:

  • Import the Redpanda Load Factor dashboard into Grafana for a BYOC cluster

  • Use the load factor panels to find the resource that limits a broker

  • Troubleshoot high load to determine whether to scale the cluster or change the workload

Prerequisites

  • A BYOC cluster running Redpanda version 26.2 or later. You can find the version on your cluster’s Overview page in the Redpanda Cloud UI.

  • A Prometheus-compatible scraper that collects metrics from your cluster’s metrics endpoint, and a Grafana instance with a Prometheus data source that points at that scraper. See Configure Redpanda monitoring and Connect Grafana.

  • rpk version 26.2.2 or later, to generate the dashboard with rpk generate grafana-dashboard. If you have an earlier version of rpk, download the dashboard JSON directly instead (see Import the dashboard).

What the dashboard measures

Utilization metrics tell you how busy a resource is, but not how much capacity remains. The dashboard closes that gap by dividing utilization by what the host can deliver. It uses two kinds of metrics, both available from your cluster’s metrics endpoint:

  • Instance capacity metrics (the redpanda_instance_* family) report the vendor’s nominal figures for the cloud instance type each broker runs on, including vCPUs, memory, disk read and write IOPS, and network throughput.

  • Host disk metrics (the redpanda_host_diskstats_* and redpanda_io_queue_config_* families) report disk reads and writes on the devices that back Redpanda’s data, and the I/O rates configured for the host’s I/O queues.

CPU, I/O scheduler, memory, and connection load factors use Redpanda’s own metrics and limits. Disk IOPS and network load factors depend on the instance capacity metrics.

Two of these limits are set below the full resource. The memory load factor reaches 100% when available memory falls to 10% of the broker’s total memory, not when memory is exhausted. The connections load factor compares active Kafka connections with the lower of 1,000 connections per GiB of memory or 3,000 connections per shard.

The dashboard computes a load factor for each resource and highlights the highest one, which is the resource that saturates first.

Redpanda does not emit a single load_factor metric. The dashboard computes the load factor from the metric families above.

Limitations

  • Redpanda detects the instance type only on AWS, so disk IOPS and network load factors are not available on GCP and Azure. On GCP and Azure BYOC clusters, the disk read IOPS, disk write IOPS, and network receive and transmit load factor panels show no data. The CPU, I/O scheduler, memory, and connection load factors work on all three cloud providers.

  • On AWS, if the instance type is not in Redpanda’s capacity table, Instance Info shows the instance type but no capacity figures. The disk IOPS and network load factor panels then show no data.

  • Disk and network capacity are the vendor’s published figures for the instance type. The capacity actually available to Redpanda can differ.

  • The Highest Load Factor panel takes the maximum of the load factor panels that have data. On GCP and Azure, and on AWS instance types without capacity figures, it does not reflect disk IOPS or network load.

Import the dashboard

  1. Generate the dashboard JSON with rpk:

    rpk generate grafana-dashboard --dashboard load-factor > redpanda-load-factor.json

    rpk downloads the dashboard from the Redpanda observability repository. It does not connect to your cluster. To skip rpk, download Redpanda-Load-Factor-Dashboard.json directly.

  2. In Grafana, go to Dashboards, click New, and select Import.

  3. Upload the JSON file, select the Prometheus data source that points at your scraper, and click Import.

The dashboard appears as Redpanda - Load Factor (Public Metrics). To confirm that data is flowing, check that Host Metrics Info shows no warning. On AWS, Instance Info also shows your instance type. On GCP and Azure, Instance Info shows the provider as unknown, which is expected.

Read the dashboard

The top row shows how close each broker is to its first bottleneck:

  • The load factor panels (Reactor / CPU, IO Scheduler, Disk Read IOPS, Disk Write IOPS, Available Memory, Connections, Net RX, and Net TX) show the current load factor for each resource, as a percentage of capacity.

  • Highest Load Factor shows the highest load factor across those panels.

The information panels describe what the estimates are based on:

  • Instance Info: the detected cloud provider and instance type, and whether capacity figures are available for it. On GCP and Azure, the panel shows the provider as unknown.

  • Host Metrics Info: whether host disk metrics are reaching the dashboard. If this panel warns that host_metrics_info is missing, confirm that the cluster runs Redpanda 26.2 or later and that your scraper is collecting from the cluster’s metrics endpoint.

  • IO Properties: the I/O rates configured for the host’s I/O queues.

The remaining panels break each load factor down over time, by broker and by shard, so you can see where load comes from: reactor and CPU utilization, I/O queue consumption, disk reads and writes, memory, connections, and network traffic.

When the load factor is high

Before you change the cluster, find out what is driving the load:

  1. In Highest Load Factor and the load factor panels, note which resource is highest. That resource is the one to act on.

  2. Compare brokers in the by-instance panels and shards in the by-shard panels. If one broker or shard runs much hotter than the others, the load is likely concentrated on a few partitions or partition leaders. Adding brokers may not help. Instead, check how partitions of your busiest topics are distributed, and consider adding partitions to spread the load.

  3. If load is spread evenly across brokers and stays high as traffic grows, the cluster needs more capacity. See Scale a BYOC Cluster.

To get alerts when load rises, create alerts on the load factor queries in Grafana or in the alerting tool that reads from your scraper. Choose thresholds based on your workload’s latency and stability requirements. Some workloads run close to 100% without problems, while others need more headroom.

Next steps