Streaming

Monitor a Stretch Cluster in Kubernetes

Detect when the operator control plane behind a Stretch Cluster loses quorum, when a member cluster becomes unreachable, and when reconciliation stalls, before those failures reach your brokers.

After reading this page, you will be able to:

  • Configure the ServiceMonitor, PrometheusRule, and source-cluster label for each multicluster operator

  • Choose a Prometheus topology for the operator metrics

  • Deploy Grafana with the prebuilt stretch cluster dashboard

Monitor a Stretch Cluster at two levels:

  • Broker metrics: The brokers export the same public metrics as any other Redpanda deployment. See Monitor Redpanda in Kubernetes.

  • Operator metrics: Each multicluster operator exports Prometheus metrics for the operator Raft group, StretchCluster member status, and reconcile health. Redpanda provides a prebuilt Grafana dashboard for these metrics, which you can generate with rpk generate grafana-dashboard.

This page covers the operator metrics. Monitor the brokers the same way you monitor any other Kubernetes deployment.

Prerequisites

  • A deployed Stretch Cluster: See Deploy a Stretch Cluster on Kubernetes.

  • Prometheus Operator: The Prometheus Operator installed on each Kubernetes cluster whose operator you want to scrape, so that the ServiceMonitor and PrometheusRule resources are reconciled.

  • Redpanda Operator v26.2.3 or later on each Kubernetes cluster. Earlier releases render the ServiceMonitor, but a multicluster operator serves its metrics endpoint over plain HTTP while the ServiceMonitor scrapes it over HTTPS, so every scrape fails during the TLS handshake and Prometheus collects no operator metrics. For the symptoms, see Troubleshoot missing operator metrics.

Scrape the operator metrics

Install or upgrade each operator Helm release with monitoring enabled so that the chart renders a ServiceMonitor for the operator’s metrics endpoint:

helm upgrade <cluster-1-name> redpanda/operator \
  --kube-context <cluster-1-context> \
  --namespace redpanda \
  --version v26.2.3 \
  --reuse-values \
  --set monitoring.enabled=true
Enabling monitoring changes the operator Deployment, which restarts the operator Pod. Upgrade one Kubernetes cluster at a time, waiting for the Raft group to report healthy before moving to the next, as described in Upgrade the operators. Restarting two of three operators at once drops quorum and stalls reconciliation.

The rendered ServiceMonitor scrapes the operator’s https port (8443) at /metrics over HTTPS. The operator terminates TLS with a self-signed certificate, so the ServiceMonitor skips certificate verification, and it authenticates with a bearer token. The endpoint stays behind Kubernetes authentication and authorization.

Passing --metrics-secure=false to the operator through the chart’s additionalCmdFlags value returns the endpoint to plain HTTP, but the chart always renders the ServiceMonitor with scheme: https. Choose plain HTTP only if you scrape the operator through your own Prometheus configuration instead of the chart’s ServiceMonitor.

To also create a PrometheusRule with the recommended recording rules and alerts for the operator’s reconcile health, set monitoring.rulesEnabled=true.

The chart labels the ServiceMonitor with app.kubernetes.io/name: operator, which comes from the chart name rather than the release name, so one selector matches it in every Kubernetes cluster. A Prometheus that already scrapes your brokers can select the operator as well, which saves running a second instance for it. Because matchLabels cannot match two values, select both resources with a matchExpressions clause:

serviceMonitorSelector:
  matchExpressions:
  - key: app.kubernetes.io/name
    operator: In
    values: [operator, redpanda] (1)
1 The operator chart’s ServiceMonitor and the Redpanda chart’s ServiceMonitor. Both labels come from their chart names, so neither changes per release. For the broker side of this, see Monitor Redpanda in Kubernetes.

Monitor each cluster on its own Prometheus

One Prometheus per Kubernetes cluster, each scraping only its local operator, is a complete setup on its own. It needs nothing beyond Scrape the operator metrics: each cluster’s Prometheus holds that operator’s series and evaluates that cluster’s copy of the PrometheusRule against them.

An operator reports on the whole deployment rather than only on itself. The operator_multicluster_raft_* series carry a label per peer and the operator_stretchcluster_* series carry a label per member, so a single cluster’s Prometheus already answers which operator holds Raft leadership, which peer is behind, and which member is unreachable, as that operator sees it.

One Prometheus also covers everything else that operator manages. An operator can run standalone Redpanda clusters alongside the StretchCluster, and its own metrics do not change either way, because the controller_runtime_* series are labeled by controller rather than by managed resource. To scrape those clusters' brokers from the same Prometheus, see Monitor Redpanda in Kubernetes.

Keeping each operator’s series in the cluster that scraped them keeps them out of your cross-region traffic entirely, which matters most when that same Prometheus also scrapes the brokers of every cluster the operator manages. For the operator metrics on their own, the saving is nominal, because an operator exports a modest and roughly fixed set of series. Weigh it against the single view that aggregation gives you.

Leave monitoring.clusterLabel at its default here. Each Prometheus holds exactly one operator’s series, so there is nothing to disambiguate, and turning the label on stamps it onto every series that operator exports, which breaks any dashboard or recording rule that matches on an exact label set:

operator-values.yaml
monitoring:
  enabled: true
  rulesEnabled: true
  clusterLabel:
    enabled: false (1)
    name: "" (2)
    value: ""
1 The default. Turn it on only when several operators' series land in one backend, as described in Label each series with its source cluster.
2 Both fields stay empty while the label is disabled, because the chart reads them only when enabled is true.

The install command in Scrape the operator metrics already produces these values: it sets monitoring.enabled and leaves monitoring.clusterLabel alone.

To read the deployment from each operator in turn, add every cluster’s Prometheus to Grafana as its own data source and switch between them with the dashboard’s DS_PROMETHEUS variable. Leave its Kubernetes cluster picker at All: each Prometheus holds one operator’s series, so the data source is what selects the cluster.

Choose this topology when you already run Prometheus in each Kubernetes cluster and want operator alerting alongside your existing per-cluster monitoring. Two limits come with it:

  • When an operator or its Prometheus is unreachable, you lose that operator’s view completely, including what it reported about the peers that are still healthy.

  • No panel compares what the operators report independently. Operators can disagree, during a leadership election for example, and noticing that means switching data sources.

To remove both limits, aggregate the metrics instead.

Aggregate metrics across clusters

Aggregating gives the dashboard one data source holding every operator’s series at once, so its panels compare all clusters in a single view, and a cluster whose operator has gone quiet stays visible beside the others. The scrape topology is unchanged: each cluster’s Prometheus still scrapes only its local operator, then forwards those series onward.

Forward them into one Prometheus-compatible backend such as Thanos, Mimir, or a dedicated aggregation Prometheus. Configure each cluster’s Prometheus to remote write to it, as in Remote write to a central Prometheus.

The cost is one more component to run, plus cross-region traffic that is nominal for these metrics. An operator’s series scale with its controllers, peers, and StretchCluster members rather than with topics and partitions, so it exports a small and roughly fixed set however much data flows through the brokers it manages. Broker metrics are the voluminous ones, which is why the example in Remote write to a central Prometheus forwards only the operator series and leaves the broker series in each cluster’s own Prometheus.

To size the forwarded volume before you commit to it, count the series that the filter selects against one cluster’s own Prometheus:

count({__name__=~"operator_.*|controller_runtime_.*|workqueue_.*"})

Label each series with its source cluster

Once several clusters write into one backend, nothing in a scraped series names the Kubernetes cluster that produced it. The scrape adds job, service, pod, and namespace labels, but those describe the Helm release and the operator Pod, so a query that compares clusters has to group by release-derived labels and map each one back to a cluster by hand. Two clusters that install the operator under the same release name produce series you cannot tell apart at all.

In Redpanda Operator v26.2.3 and later, enable monitoring.clusterLabel on each operator release to stamp the source cluster onto every series that operator exports:

helm upgrade <cluster-1-name> redpanda/operator \
  --kube-context <cluster-1-context> \
  --namespace redpanda \
  --version v26.2.3 \
  --reuse-values \
  --set monitoring.enabled=true \
  --set monitoring.clusterLabel.enabled=true

The chart adds a relabeling to the ServiceMonitor endpoint that sets the redpanda_k8s_cluster label to the value of multicluster.name, which is the cluster name that rpk k8s multicluster status reports. Because each cluster installs the operator with its own multicluster.name, enabling the label is the only change each cluster needs.

With the label on, the stretch cluster dashboard’s Kubernetes cluster picker lists its values, so you can show one cluster or compare them. Group and filter on it in your own queries the same way:

sum by (redpanda_k8s_cluster) (rate(controller_runtime_reconcile_total[5m]))

To fit an existing labeling convention, override the label name, its value, or both:

monitoring:
  enabled: true
  clusterLabel:
    enabled: true
    name: k8s_cluster (1)
    value: prod-us-east-1 (2)
1 The label to set. Leave it empty to use redpanda_k8s_cluster. The stretch cluster dashboard’s Kubernetes cluster picker queries the default name, so if you override it, rename the label in that picker’s query and in the panel matchers as well.
2 The label’s value. Leave it empty to inherit multicluster.name. Set it explicitly when multicluster.enabled is false, such as when you aggregate metrics from independent single-cluster operator installations. If both this value and multicluster.name are empty, the chart fails to render rather than stamp an empty label.

The label is off by default, so turning it on adds a label to series that did not carry one. Update any dashboard or recording rule that matches on an exact label set before you roll it out.

The recording rules that monitoring.rulesEnabled creates keep only the controller or workqueue name, job, and namespace, and drop every other label, including this one. Recorded series such as operator:reconcile_rate:5m, and the alerts that evaluate them, still combine all clusters. Alerts that evaluate raw metrics, such as StretchClusterMemberUnreachable, keep the label.

Remote write to a central Prometheus

This example aggregates the operator metrics from three Kubernetes clusters into one Prometheus instance. Each cluster keeps its own Prometheus, which scrapes its local operator and forwards those series to a central Prometheus that accepts remote write. The stretch cluster dashboard then reads from the central instance.

  1. On the cluster that hosts the aggregated backend, deploy a Prometheus that accepts remote write:

    aggregation-prometheus.yaml
    apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: aggregation
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  enableRemoteWriteReceiver: true (1)
    1 Serves the remote write endpoint at /api/v1/write. Prometheus documents this receiver as suited to low-volume ingestion rather than as a replacement for scraping, which fits here because each cluster forwards only its operator series.

  2. On each Kubernetes cluster, authorize the Prometheus ServiceAccount to read the operator’s metrics endpoint. The operator chart creates the ClusterRole but binds only its own ServiceAccount to it:

    metrics-reader-binding.yaml
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-metrics-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: <cluster-1-name>-redpanda-metrics-reader (1)
subjects:
- kind: ServiceAccount
  name: prometheus (2)
  namespace: monitoring
    1 The ClusterRole that the operator chart creates, named for the operator release and its namespace. To find it, run kubectl get clusterrole | grep metrics-reader.
    2 The ServiceAccount that your Prometheus Pods run as. Without this binding, every scrape of the operator returns 403.

  3. On each Kubernetes cluster, deploy a Prometheus that scrapes the local operator and forwards its series:

    cluster-prometheus.yaml
    apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: redpanda
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: redpanda (1)
  serviceMonitorSelector:
    matchLabels:
      app.kubernetes.io/name: operator (2)
  ruleNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: redpanda
  ruleSelector:
    matchLabels:
      app.kubernetes.io/name: operator (3)
  remoteWrite:
  - url: https://<aggregation-prometheus-address>/api/v1/write (4)
    writeRelabelConfigs:
    - sourceLabels: [__name__] (5)
      regex: (operator_.*|controller_runtime_.*|workqueue_.*|up)
      action: keep
    1 The namespace that holds the operator release. Kubernetes sets this label on every namespace.
    2 The label that the operator chart puts on its ServiceMonitor. The value is the chart name, so it is the same in every cluster. To select on something else, add your own labels with the chart’s monitoring.labels value.
    3 Selects the PrometheusRule that monitoring.rulesEnabled creates, which carries the same label. Omit both rule selectors if you do not use that value.
    4 The address you exposed the aggregation Prometheus on. Add basicAuth, authorization, or tlsConfig to this entry to authenticate.
    5 Forwards the operator series and the up series for its targets, so broker metrics stay in the local Prometheus instead of crossing regions. Remove the writeRelabelConfigs block to forward everything this Prometheus scrapes.

  4. Query the aggregation Prometheus to confirm that it holds series from every cluster:

    count by (redpanda_k8s_cluster) (up)

    The result has one row per Kubernetes cluster. A missing row means that cluster’s Prometheus either is not scraping its operator or is not reaching the receiver, so check its target status and its remote write queue metrics.

A remote write receiver accepts samples from anything that can reach it. The Prometheus Operator creates a prometheus-operated Service for the aggregation instance. When you expose that Service to the other clusters through an Ingress or a load balancer, require TLS and authentication on the route, and set basicAuth, authorization, or tlsConfig on each cluster’s remoteWrite entry to match.

The externalLabels field on each cluster’s Prometheus is an alternative to monitoring.clusterLabel. Prometheus adds those labels when it talks to external systems, so they reach the aggregated backend but are absent from the cluster’s own Prometheus. The chart value stamps the label at scrape time instead, so the same label is present in both places.

This filter deliberately excludes the recorded series that monitoring.rulesEnabled creates, such as operator:reconcile_rate:5m. Those keep only the controller or workqueue name, job, and namespace, so they carry nothing that identifies a cluster and would collide in the aggregated backend. Leave the rules enabled in each cluster, where they evaluate against that cluster’s own data.

Generate the stretch cluster dashboard

Generate the stretch cluster observability dashboard and save it to a file:

rpk generate grafana-dashboard --dashboard operations-stretch > stretch-cluster-dashboard.json

The dashboard selects its Prometheus data source through a DS_PROMETHEUS dashboard variable, so it imports into any Grafana instance without modification. Its rows cover:

  • Multicluster raft (operator_multicluster_raft_*): Operator Raft leader and term, leader changes, per-peer send latency and queue length, send errors, follower lag, peer reachability, and leadership history.

  • StretchCluster member status (operator_stretchcluster_*): Per-member reachability, broker shortfall (desired versus ready brokers), spec drift between clusters, and replication health.

  • Reconcile activity (controller_runtime_reconcile_*): Reconcile rate, error rate, and duration per controller, plotted against the steady-state rate that operator_controller_reconcile_steady_state_total reports.

  • Queues and workers (workqueue_*, controller_runtime_active_workers): Workqueue depth and retry rate, and how close each controller runs to its concurrency limit.

  • Reconcile-health signals (operator_controller_reconcile_last_success_timestamp_seconds): How long ago each controller last reached steady state, as the Raft leader sees it.

Three pickers scope what those panels show:

  • Namespace and Job choose which operator installs to include. Their values come from a metric that only the Redpanda Operator exports, so even All stays within Redpanda Operator installs. That scoping matters because controller_runtime_* and workqueue_* are not Redpanda-specific: every controller-runtime operator on the cluster exports them under the same names, so on a Prometheus that scrapes the whole cluster, unscoped reconcile and queue panels also plot Karpenter, Flux, or cert-manager.

  • Kubernetes cluster filters on the redpanda_k8s_cluster label described in Label each series with its source cluster. It defaults to .*, so it changes nothing until you enable that label.

To import stretch-cluster-dashboard.json into an existing Grafana instance, use Dashboards > Import. To deploy Grafana with the dashboard already provisioned, see Deploy Grafana with the dashboard.

Deploy Grafana with the dashboard

This example deploys Grafana with the Grafana Helm chart, preprovisioned with a Prometheus data source and the generated stretch cluster dashboard. Run these commands against the cluster where your Prometheus backend is reachable.

  1. Create the monitoring namespace and a ConfigMap containing the generated dashboard:

    kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -

kubectl create configmap redpanda-stretch-dashboard \
  --namespace monitoring \
  --from-file=stretch-cluster-dashboard.json

  2. Save the following Helm values to grafana-values.yaml:

    datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://<prometheus-service>:9090 (1)
        isDefault: true
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: redpanda
        folder: Redpanda
        type: file
        options:
          path: /var/lib/grafana/dashboards/redpanda
dashboardsConfigMaps:
  redpanda: redpanda-stretch-dashboard (2)
    1 The address of the Prometheus that holds the operator metrics. With one Prometheus per Kubernetes cluster, add an entry per cluster, each with its own name, so the dashboard’s DS_PROMETHEUS variable can switch between them.
    2 The ConfigMap created in the previous step.

  3. Install Grafana:

    helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install grafana grafana/grafana \
  --namespace monitoring \
  --values grafana-values.yaml

  4. Get the admin password and open Grafana:

    kubectl get secret grafana --namespace monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

kubectl port-forward service/grafana 3000:80 --namespace monitoring

    Log in at http://localhost:3000 as admin with the retrieved password. The stretch cluster dashboard is in the Redpanda folder.

Troubleshoot missing operator metrics

If the operator’s Prometheus target reports as down, or no operator_* series reach your backend, check the following:

  • Operator version: On releases earlier than v26.2.3, a multicluster operator serves its metrics endpoint over plain HTTP while the ServiceMonitor scrapes it over HTTPS, so the scrape fails during the TLS handshake and collects nothing. Upgrade each operator to v26.2.3 or later, one Kubernetes cluster at a time.

  • A leftover scheme override: If you patched the rendered ServiceMonitor to scheme: http to work around that behavior, remove the patch. The endpoint serves TLS in v26.2.3 and later, and an HTTP scrape of it fails.

  • Authorization: The metrics endpoint requires a bearer token that is authorized for the /metrics non-resource URL. The chart creates a ClusterRole for this (for example, <cluster-1-name>-redpanda-metrics-reader) and binds the operator’s own ServiceAccount to it. If Prometheus scrapes under a different ServiceAccount, bind that ServiceAccount to the same ClusterRole, or the scrape fails with a 403 response.

To capture the operator’s metrics for a support case, run rpk k8s multicluster bundle. It samples /metrics from every Kubernetes cluster in the deployment, and it detects the scheme each operator serves, so one bundle collects metrics from operators on either side of this change.

References