You can monitor the health of your Redpanda managed connectors with metrics that Redpanda exports through a Prometheus HTTPS endpoint. You can use Grafana to visualize the metrics and set up alerts.
The connectors dashboard renders metrics that are exported by managed connectors. However, when a connector does not create a task (for example, an empty topic list), the dashboard will not show metrics for that connector.
Prometheus is an open-source system monitoring and alerting tool that collects and stores metrics as time-series data identified by a metric name and key/value pairs.
|You can quickly get Prometheus and Grafana running locally, but not for production instances. For production instances, deploy Prometheus and Grafana as a standalone or managed service, as described below.|
To configure and use Prometheus to monitor Redpanda managed connector metrics:
In Redpanda Cloud, go to Overview > *How to connect > Prometheus. Click the Copy icon for Prometheus YAML to copy its content into your clipboard.
prometheus.ymlfile in the Prometheus root folder to add the Redpanda configuration under
scrape_configs: - job_name: redpandaCloud static_configs: - targets: - ... metrics_path: /api/cloud/prometheus/public_metrics basic_auth: username: prometheus password: ... scheme: https
Save the configuration file, and restart Prometheus to apply changes.
Observe in Prometheus that metrics from Redpanda endpoints are scraped.
You can monitor the following metrics for your Redpanda managed connectors.
Number of tasks for a specific connector, grouped by status:
running - Tasks that are healthy and running.
paused - Tasks that were paused by a user request.
failed - Tasks that failed during execution.
paused tasks. Create an alert for failed
The number of records still to be processed by a connector. This metric
is emitted for sink connectors only (
For newly-created connectors, the metric is high until the connector sinks all historical data.
Expect the lag not to increase over time.
Age of the last record written to the target cluster by the MirrorMaker 2 connector. This metric is emitted for each partition.
For newly-created connectors, the metric is high until the connector processes all historical data.
Expect the latency to not increase over time.
Count of records sent to the cluster by source connectors for each topic.
record errors- Total number of record errors seen in connector tasks.
record failures- Total number of record failures seen in connector tasks.
record skipped- Total number of records skipped by connector tasks.
record sent- Total number of records sent by connector producers.
record retry- Total number of records sent retries by connector producers.