Streaming

Monitor a Stretch Cluster in Kubernetes

Detect when the operator control plane behind a Stretch Cluster loses quorum, when a member cluster becomes unreachable, and when reconciliation stalls, before those failures reach your brokers.

After reading this page, you will be able to:

  • Configure a ServiceMonitor and PrometheusRule for each multicluster operator

  • Aggregate operator metrics from all Kubernetes clusters into one Prometheus backend

  • Deploy Grafana with the prebuilt stretch cluster dashboard

Monitor a Stretch Cluster at two levels:

  • Broker metrics: The brokers export the same public metrics as any other Redpanda deployment. See Monitor Redpanda in Kubernetes.

  • Operator metrics: Each multicluster operator exports Prometheus metrics for the operator Raft group, StretchCluster member status, and reconcile health. Redpanda provides a prebuilt Grafana dashboard for these metrics, which you can generate with rpk generate grafana-dashboard.

This page covers the operator metrics. Monitor the brokers the same way you monitor any other Kubernetes deployment.

Prerequisites

  • A deployed Stretch Cluster: See Deploy a Stretch Cluster on Kubernetes.

  • Prometheus Operator: The Prometheus Operator installed on each Kubernetes cluster whose operator you want to scrape, so that the ServiceMonitor and PrometheusRule resources are reconciled.

  • Redpanda Operator v26.2.1 or later on each Kubernetes cluster, if you want the PrometheusRule described in Scrape the operator metrics. The monitoring.enabled value works on earlier 26.2 releases.

Scrape the operator metrics

Install or upgrade each operator Helm release with monitoring enabled so that the chart renders a ServiceMonitor for the operator’s metrics endpoint:

helm upgrade <cluster-1-name> redpanda/operator \
  --kube-context <cluster-1-context> \
  --namespace redpanda \
  --version {latest-operator-version} \
  --reuse-values \
  --set monitoring.enabled=true
Enabling monitoring changes the operator Deployment, which restarts the operator Pod. Upgrade one Kubernetes cluster at a time, waiting for the Raft group to report healthy before moving to the next, as described in Upgrade the operators. Restarting two of three operators at once drops quorum and stalls reconciliation.

To also create a PrometheusRule with the recommended recording rules and alerts for the operator’s reconcile health, set monitoring.rulesEnabled=true. This value requires Redpanda Operator v26.2.1 or later.

Aggregate metrics across clusters

The stretch cluster dashboard reads from a single Prometheus data source, so aggregate the operator metrics from all Kubernetes clusters into one Prometheus-compatible backend. For example, configure each cluster’s Prometheus to remote write to a central instance such as Thanos, Mimir, or a dedicated aggregation Prometheus.

Without this, the dashboard shows only the cluster whose Prometheus you point it at, which defeats the purpose of the multicluster panels.

Generate the stretch cluster dashboard

Generate the stretch cluster observability dashboard and save it to a file:

rpk generate grafana-dashboard --dashboard operations-stretch > stretch-cluster-dashboard.json

The dashboard selects its Prometheus data source through a DS_PROMETHEUS dashboard variable, so it imports into any Grafana instance without modification. Its panels cover:

  • Multicluster Raft (operator_multicluster_raft_*): Operator Raft leader and term, leader changes, per-peer send latency and queue length, send errors, follower lag, peer reachability, and leadership history.

  • StretchCluster member status (operator_stretchcluster_*): Per-member reachability, broker shortfall (desired versus ready brokers), spec drift between clusters, and replication health.

  • Reconcile health (operator_controller_*): Reconcile rates, error rates, and duration per controller, plus workqueue depth, worker saturation, and retry rates.

You can either import stretch-cluster-dashboard.json into an existing Grafana instance through Dashboards > Import, or deploy Grafana with the dashboard preprovisioned as shown in the next section.

Deploy Grafana with the dashboard

This example deploys Grafana with the Grafana Helm chart, preprovisioned with a Prometheus data source and the generated stretch cluster dashboard. Run these commands against the cluster where your aggregated Prometheus backend is reachable.

  1. Create the monitoring namespace and a ConfigMap containing the generated dashboard:

    kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -

kubectl create configmap redpanda-stretch-dashboard \
  --namespace monitoring \
  --from-file=stretch-cluster-dashboard.json

  2. Save the following Helm values to grafana-values.yaml:

    datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://<prometheus-service>:9090 (1)
        isDefault: true
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: redpanda
        folder: Redpanda
        type: file
        options:
          path: /var/lib/grafana/dashboards/redpanda
dashboardsConfigMaps:
  redpanda: redpanda-stretch-dashboard (2)
    1 The address of the Prometheus instance that aggregates the operator metrics from all clusters.
    2 The ConfigMap created in the previous step.

  3. Install Grafana:

    helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install grafana grafana/grafana \
  --namespace monitoring \
  --values grafana-values.yaml

  4. Get the admin password and open Grafana:

    kubectl get secret grafana --namespace monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

kubectl port-forward service/grafana 3000:80 --namespace monitoring

    Log in at http://localhost:3000 as admin with the retrieved password. The stretch cluster dashboard is in the Redpanda folder.

References