Monitor Redpanda Cloud

You can configure monitoring on your Redpanda Dedicated or BYOC cluster to maintain system health and optimize performance. Redpanda Cloud exports metrics for all brokers and Kafka Connect workers from a single Prometheus endpoint. This endpoint can be found on the Overview page for your cluster, under How to connect.

You can monitor Redpanda with Prometheus or with any other monitoring and alerting tool, such as Datadog, New Relic, Elastic Cloud, Google, or Azure.

To maximize performance, Redpanda exports some metrics only when the underlying feature is in use. For example, a metric for consumer groups, redpanda_kafka_consumer_group_committed_offset, is only exported when groups are registered.

Configure Prometheus

To monitor a Dedicated or BYOC cluster in Prometheus:

  1. On the Redpanda Cloud Overview page for your cluster, under How to connect, click the Prometheus tab.

  2. Click the copy icon for Prometheus YAML to copy the contents to your clipboard.

    How to connect screenshot

    The YAML contains the Prometheus scrape target configuration, as well as authentication, for the cluster.

    - job_name: redpandaCloud-sample
  static_configs:
    - targets:
        - console-<id>.<identifier>.fmc.cloud.redpanda.com
  metrics_path: /api/cloud/prometheus/public_metrics
  basic_auth:
    username: prometheus
    password: "<prom_pass>"
  scheme: https

  3. Save this information to Prometheus, and observe in Prometheus that metrics from Redpanda endpoints are scraped.

Configure Datadog

To monitor a Dedicated or BYOC cluster in Datadog:

  1. On the Redpanda Cloud Overview page for your cluster, under How to connect, click the Prometheus tab.

  2. Click the copy icon for Prometheus YAML to find scrape target configuration, as well as authentication, for the cluster.

  3. In Datadog, define the openmetrics_endpoint URL for that monitored cluster. The integration configuration should look similar to the following:

    instances:
  # The endpoint to collect metrics about Polaris. Replace ORGANIZATION_NAME with the name of your organization.
  - openmetrics_endpoint: https://console-<id>.<identifier>.fmc.cloud.redpanda.com/api/cloud/prometheus/public_metrics
    use_openmetrics: true
    collect_counters_with_distributions: true

    auth_type: basic
    username: prom_user
    password: prom_pass

  4. Restart the Datadog agent.

Because the Prometheus endpoint in Redpanda Cloud aggregates metrics for all cluster services, only a single Datadog agent is required. The agent must run in a container in your own container infrastructure. Redpanda does not support launching this container inside a Dedicated or BYOC Kubernetes cluster. For more information, see the Datadog documentation and Redpanda Datadog integration.

Use Redpanda monitoring examples

For hands-on learning, Redpanda provides a repository with examples of monitoring Redpanda with Prometheus and Grafana: redpanda-data/observability.

Example Redpanda Ops Dashboard^

It includes example Grafana dashboards and a sandbox environment in which you launch a Dockerized Redpanda cluster and create a custom workload to monitor with dashboards.

Monitor for performance and health

This section provides guidelines and example queries using Redpanda’s public metrics to optimize your system’s performance and monitor its health.

To help detect and mitigate anomalous system behaviors, capture baseline metrics of your healthy system at different stages (at start-up, under high load, in steady state) so you can set thresholds and alerts according to those baselines.

Redpanda architecture

Understanding the unique aspects of Redpanda’s architecture and data path can improve your performance, debugging, and tuning skills:

  • Redpanda replicates partitions across brokers in a cluster using Raft, where each partition is a Raft consensus group. A message written from the Kafka API flows down to the Raft implementation layer that eventually directs it to a broker to be stored. Metrics about the Raft layer can reveal the health of partitions and data flowing within Redpanda.

  • Redpanda is designed with a thread-per-core model that it implements with the Seastar library. With each application thread pinned to a CPU core, when observing or analyzing the behavior of a specific application, monitor the relevant metrics with the label for the specific shard, if available.

Infrastructure resources

The underlying infrastructure of your system should have sufficient margins to handle peaks in processing, storage, and I/O loads. Monitor infrastructure health with the following queries.

CPU usage

For the total CPU uptime, monitor redpanda_uptime_seconds_total. Monitoring its rate of change with the following query can help detect unexpected dips in uptime:

rate(redpanda_uptime_seconds_total[5m])

For the total CPU busy (non-idle) time, monitor redpanda_cpu_busy_seconds_total.

To detect unexpected idling, you can query the rate of change as a percentage of the shard that is in use at a given point in time.

rate(redpanda_cpu_busy_seconds_total[5m])

While CPU utilization at the host-level might appear high (for example, 99-100% utilization) when I/O events like message arrival occur, the actual Redpanda process utilization is likely low. System-level metrics such as those provided by the top command can be misleading.

This high host-level CPU utilization happens because Redpanda uses Seastar, which runs event loops on every core (also referred to as a reactor), constantly polling for the next task. This process never blocks and will increment clock ticks. It doesn’t necessarily mean that Redpanda is busy.

Use redpanda_cpu_busy_seconds_total to monitor the actual Redpanda CPU utilization. When it indicates close to 100% utilization over a given period of time, make sure to also monitor produce and consume latency as they may then start to increase as a result of resources becoming overburdened.

Memory allocated

To monitor the percentage of memory allocated, use a formula with redpanda_memory_allocated_memory and redpanda_memory_free_memory:

sum(redpanda_memory_allocated_memory) / (sum(redpanda_memory_free_memory) + sum(redpanda_memory_allocated_memory))

Disk used

To monitor the percentage of disk consumed, use a formula with redpanda_storage_disk_free_bytes and redpanda_storage_disk_total_bytes:

1 - (sum(redpanda_storage_disk_free_bytes) / sum(redpanda_storage_disk_total_bytes))

Also monitor redpanda_storage_disk_free_space_alert for an alert when available disk space is low or degraded.

IOPS

For read and write I/O operations per second (IOPS), monitor the redpanda_io_queue_total_read_ops and redpanda_io_queue_total_write_ops counters:

rate(redpanda_io_queue_total_read_ops[5m]),
rate(redpanda_io_queue_total_write_ops[5m])

Throughput

While maximizing the rate of messages moving from producers to brokers then to consumers depends on tuning each of those components, the total throughput of all topics provides a system-level metric to monitor. When you observe abnormal, unhealthy spikes or dips in producer or consumer throughput, look for correlation with changes in the number of active connections (redpanda_rpc_active_connections) and logged errors to drill down to the root cause.

The total throughput of a cluster can be measured by the producer and consumer rates across all topics.

To observe the total producer and consumer rates of a cluster, monitor redpanda_kafka_request_bytes_total with the produce and consume labels, respectively.

Producer throughput

For the produce rate, create a query to get the produce rate across all topics:

sum(rate(redpanda_kafka_request_bytes_total{redpanda_request="produce"} [5m] )) by (redpanda_request)

Consumer throughput

For the consume rate, create a query to get the total consume rate across all topics:

sum(rate(redpanda_kafka_request_bytes_total{redpanda_request="consume"} [5m] )) by (redpanda_request)

Latency

Latency should be consistent between produce and fetch sides. It should also be consistent over time. Take periodic snapshots of produce and fetch latencies, including at upper percentiles (95%, 99%), and watch out for significant changes over a short duration.

In Redpanda, the latency of produce and fetch requests includes the latency of inter-broker RPCs that are born from Redpanda’s internal implementation using Raft.

Kafka consumer latency

To monitor Kafka consumer request latency, use the redpanda_kafka_request_latency_seconds histogram with the label redpanda_request="consume". For example, create a query for the 99th percentile:

histogram_quantile(0.99, sum(rate(redpanda_kafka_request_latency_seconds_bucket{redpanda_request="consume"}[5m])) by (le, provider, region, instance, namespace, pod))

You can monitor the rate of Kafka consumer requests using redpanda_kafka_request_latency_seconds_count with the redpanda_request="consume" label:

rate(redpanda_kafka_request_latency_seconds_count{redpanda_request="consume"}[5m])

Kafka producer latency

To monitor Kafka producer request latency, use the redpanda_kafka_request_latency_seconds histogram with the redpanda_request="produce" label. For example, create a query for the 99th percentile:

histogram_quantile(0.99, sum(rate(redpanda_kafka_request_latency_seconds_bucket{redpanda_request="produce"}[5m])) by (le, provider, region, instance, namespace, pod))

You can monitor the rate of Kafka producer requests with redpanda_kafka_request_latency_seconds_count with the redpanda_request="produce" label:

rate(redpanda_kafka_request_latency_seconds_count{redpanda_request="produce"}[5m])

Internal RPC latency

To monitor Redpanda internal RPC latency, use the redpanda_rpc_request_latency_seconds histogram. For example, create a query for the 99th percentile latency:

histogram_quantile(0.99, (sum(rate(redpanda_rpc_request_latency_seconds_bucket[5m])) by (le, provider, region, instance, namespace, pod, redpanda_server)))

You can monitor the rate of internal RPC requests with redpanda_rpc_request_latency_seconds histogram’s count:

rate(redpanda_rpc_request_latency_seconds_count[5m])

Partition health

The health of Kafka partitions often reflects the health of the brokers that host them. Thus, when alerts occur for conditions such as under-replicated partitions or more frequent leadership transfers, check for unresponsive or unavailable brokers.

With Redpanda’s internal implementation of the Raft consensus protocol, the health of partitions is also reflected in any errors in the internal RPCs exchanged between Raft peers.

Leadership changes

Stable clusters have a consistent balance of leaders across all brokers, with few to no leadership transfers between brokers.

To observe changes in leadership, monitor the redpanda_raft_leadership_changes counter. For example, use a query to get the total rate of increase of leadership changes for a cluster:

sum(rate(redpanda_raft_leadership_changes[5m]))

Under-replicated partitions

A healthy cluster has partition data fully replicated across its brokers.

An under-replicated partition is at higher risk of data loss. It also adds latency because messages must be replicated before being committed. To know when a partition isn’t fully replicated, create an alert for the redpanda_kafka_under_replicated_replicas gauge when it is greater than zero:

redpanda_kafka_under_replicated_replicas > 0

Under-replication can be caused by unresponsive brokers. When an alert on redpanda_kafka_under_replicated_replicas is triggered, identify the problem brokers and examine their logs.

Leaderless partitions

A healthy cluster has a leader for every partition.

A partition without a leader cannot exchange messages with producers or consumers. To identify when a partition doesn’t have a leader, create an alert for the redpanda_cluster_unavailable_partitions gauge when it is greater than zero:

redpanda_cluster_unavailable_partitions > 0

Leaderless partitions can be caused by unresponsive brokers. When an alert on redpanda_cluster_unavailable_partitions is triggered, identify the problem brokers and examine their logs.

Raft RPCs

Redpanda’s Raft implementation exchanges periodic status RPCs between a broker and its peers. The redpanda_node_status_rpcs_timed_out gauge increases when a status RPC times out for a peer, which indicates that a peer may be unresponsive and may lead to problems with partition replication that Raft manages. Monitor for non-zero values of this gauge, and correlate it with any logged errors or changes in partition replication.

Consumers

Consumer group lag

When working with Kafka consumer groups, the consumer group lag—the difference between the broker’s latest (max) offset and the group’s last committed offset—is a performance indicator of how fresh the data being consumed is. While higher lag for archival consumers is expected, high lag for real-time consumers could indicate that the consumers are overloaded and thus may need their topics to be partitioned more, or to spread the load to more consumers.

To monitor consumer group lag, create a query with the redpanda_kafka_max_offset and redpanda_kafka_consumer_group_committed_offset gauges:

max by(redpanda_namespace, redpanda_topic, redpanda_partition)(redpanda_kafka_max_offset{redpanda_namespace="kafka"}) - on(redpanda_topic, redpanda_partition) group_right max by(redpanda_group, redpanda_topic, redpanda_partition)(redpanda_kafka_consumer_group_committed_offset)

Services

Monitor the health of specific Redpanda services with the following metrics.

Schema Registry

Schema Registry request latency:

histogram_quantile(0.99, (sum(rate(redpanda_schema_registry_request_latency_seconds_bucket[5m])) by (le, provider, region, instance, namespace, pod)))

Schema Registry request rate:

rate(redpanda_schema_registry_request_latency_seconds_count[5m]) + sum without(redpanda_status)(rate(redpanda_schema_registry_request_errors_total[5m]))

Schema Registry request error rate:

rate(redpanda_schema_registry_request_errors_total[5m])

REST proxy

REST proxy request latency:

histogram_quantile(0.99, (sum(rate(redpanda_rest_proxy_request_latency_seconds_bucket[5m])) by (le, provider, region, instance, namespace, pod)))

REST proxy request rate:

rate(redpanda_rest_proxy_request_latency_seconds_count[5m]) + sum without(redpanda_status)(rate(redpanda_rest_proxy_request_errors_total[5m]))

REST proxy request error rate:

rate(redpanda_rest_proxy_request_errors_total[5m])

References