# Monitor Kubernetes Shadow Links > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Monitor Kubernetes Shadow Links latest-redpanda-tag: v25.3.11 latest-console-tag: v3.7.3 latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: November 19, 2026 latest-connect-version: 4.93.0 docname: kubernetes/shadowing/k-monitor-shadowing page-component-name: streaming page-version: "25.3" page-component-version: "25.3" page-component-title: Streaming page-relative-src-path: kubernetes/shadowing/k-monitor-shadowing.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/v/25.3/modules/manage/pages/kubernetes/shadowing/k-monitor-shadowing.adoc description: Monitor shadow link health in Kubernetes using ShadowLink resources, status commands, metrics, and best practices. page-git-created-date: "2025-12-16" page-git-modified-date: "2026-01-06" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/25.3/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/25.3/get-started/licensing/#self-managed) apply. Monitor your [shadow links](https://docs.redpanda.com/streaming/25.3/manage/kubernetes/shadowing/k-shadow-linking/) to ensure proper replication performance and understand your disaster recovery readiness. For Kubernetes deployments, you can monitor shadow links using the Redpanda Operator’s `ShadowLink` resource status or by using `rpk` commands directly. > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Kubernetes Failover Runbook](https://docs.redpanda.com/streaming/25.3/manage/kubernetes/shadowing/k-failover-runbook/) for immediate step-by-step disaster procedures. ## [](#status-commands)Status commands ### Operator To list existing shadow links: ```bash kubectl get shadowlink --namespace ``` Example output NAME SYNCED link True The synced status is `True` for a healthy shadow link. If the synced status is `False`, use `kubectl describe` to investigate the issue. To view detailed shadow link status and configuration: ```bash kubectl describe shadowlink --namespace ``` Example output Name: link Namespace: redpanda-system API Version: cluster.redpanda.com/v1alpha2 Kind: ShadowLink Status: Conditions: Status: True Type: Synced Message: Shadow link is synced Shadow Topics: Name: orders State: active Name: inventory State: active Tasks: Name: Source Topic Sync State: active Name: Consumer Group Shadowing State: active Name: Security Migrator State: active The `kubectl describe` output shows: - **Shadow link state**: Overall operational state in the `Status` section - **Individual topic states**: Current state of each replicated topic under `Shadow Topics` - **Task status**: Health of replication tasks under `Tasks` - **Sync status**: Whether the resource is properly synced (`Synced: True` in conditions) - **Configuration**: Complete shadow link configuration including connection settings and filters Look for `Synced: True` in Conditions and `active` state for topics and tasks. For more detailed monitoring or troubleshooting, you can also use `rpk` commands as shown in the Helm tab. ### Helm To list existing shadow links: ```bash kubectl exec --namespace --container redpanda -- \ rpk shadow list ``` Example output NAME UID STATE disaster-recovery-link 70f25b41-9bad-4e31-9f81-d302c8676397 ACTIVE To view shadow link configuration details: ```bash kubectl exec --namespace --container redpanda -- \ rpk shadow describe ``` For detailed command options, see [`rpk shadow list`](https://docs.redpanda.com/streaming/25.3/reference/rpk/rpk-shadow/rpk-shadow-list/) and [`rpk shadow describe`](https://docs.redpanda.com/streaming/25.3/reference/rpk/rpk-shadow/rpk-shadow-describe/). This command shows the complete configuration of the shadow link, including connection settings, filters, and synchronization options. To check your shadow link status and ensure proper operation: ```bash kubectl exec --namespace --container redpanda -- \ rpk shadow status ``` Example output OVERVIEW === NAME disaster-recovery-link UID 70f25b41-9bad-4e31-9f81-d302c8676397 STATE ACTIVE TASKS === NAME BROKER\_ID SHARD STATE REASON Source Topic Sync 0 0 ACTIVE Source Topic Sync has started Consumer Group Shadowing 0 0 ACTIVE Group mirroring task finished successfully Security Migrator Task 0 0 ACTIVE Security Migrator Task has started TOPICS === Name: orders, State: ACTIVE PARTITION SRC\_LSO SRC\_HWM DST\_HWM LAG 0 1000 1234 1230 4 1 2000 2456 2450 6 Name: inventory, State: ACTIVE PARTITION SRC\_LSO SRC\_HWM DST\_HWM LAG 0 500 789 789 0 Key indicators: - **State: active**: Shadow link is replicating - **Tasks: active**: All replication tasks are running - **Lag**: Message count difference between source and shadow (lower is better) For troubleshooting specific issues, you can use command options to show individual status sections. See [`rpk shadow status`](https://docs.redpanda.com/streaming/25.3/reference/rpk/rpk-shadow/rpk-shadow-status/) for available status options. The status output includes the following: - **Shadow link state**: Overall operational state (`ACTIVE`, `PAUSED`). - **Individual topic states**: Current state of each replicated topic (`ACTIVE`, `FAULTED`, `FAILING_OVER`, `FAILED_OVER`, `PAUSED`). - **Task status**: Health of replication tasks across brokers (`ACTIVE`, `FAULTED`, `NOT_RUNNING`, `LINK_UNAVAILABLE`). For details about shadow link tasks, see [Shadow link tasks](https://docs.redpanda.com/streaming/25.3/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). - **Lag information**: Replication lag per partition showing source vs shadow high watermarks (HWM). ## [](#troubleshoot)Troubleshoot ### [](#topics-in-faulted-state)Topics in FAULTED state When monitoring shadow links, you may see topics showing `FAULTED` state in status output. Check shadow cluster logs for specific error messages: ```bash kubectl logs --namespace --container redpanda | grep -i "shadow\|error" ``` Common causes include: - Source topic deleted: topic no longer exists on source cluster - Permission denied: shadow link service account lacks required permissions - Network interruption: temporary connectivity issues If the source topic still exists and should be replicated, delete and recreate the shadow link to reset the faulted state. ### [](#high-replication-lag)High replication lag When monitoring shadow links, you may see LAG values continuously increasing in `rpk shadow status`. Check the following: - Check source cluster load: high produce rate may exceed replication capacity - Check shadow cluster resources: CPU, memory, or disk constraints - Check network bandwidth: verify sufficient bandwidth between clusters To resolve: - Scale shadow cluster resources if constrained - Verify network connectivity and bandwidth - Review topic configuration for optimization opportunities ### [](#task-shows-link_unavailable)Task shows LINK_UNAVAILABLE When monitoring shadow links, you may see tasks showing `LINK_UNAVAILABLE` state with "No brokers available" message. Common causes include: - Source cluster requires SASL authentication but shadow link not configured for it - Source cluster unreachable from shadow cluster - Network policy blocking traffic between clusters To resolve: - Verify SASL configuration if source cluster requires authentication - Test network connectivity: `kubectl exec` into shadow pod and try connecting to source cluster - Check Kubernetes NetworkPolicies and firewall rules ## [](#shadow-link-metrics)Metrics Shadowing provides comprehensive metrics to track replication performance and health with the [`public_metrics`](https://docs.redpanda.com/streaming/25.3/reference/public-metrics-reference/) endpoint. | Metric | Type | Description | | --- | --- | --- | | redpanda_shadow_link_shadow_lag | Gauge | The lag of the shadow partition against the source partition, calculated as source partition LSO (Last Stable Offset) minus shadow partition HWM (High Watermark). Monitor by shadow_link_name, topic, and partition to understand replication lag for each partition. | | redpanda_shadow_link_total_bytes_fetched | Count | The total number of bytes fetched by a sharded replicator (bytes received by the client). Labeled by shadow_link_name and shard to track data transfer volume from the source cluster. | | redpanda_shadow_link_total_bytes_written | Count | The total number of bytes written by a sharded replicator (bytes written to the write_at_offset_stm). Uses shadow_link_name and shard labels to monitor data written to the shadow cluster. | | redpanda_shadow_link_client_errors | Count | The number of errors seen by the client. Track by shadow_link_name and shard to identify connection or protocol issues between clusters. | | redpanda_shadow_link_shadow_topic_state | Gauge | Number of shadow topics in the respective states. Labeled by shadow_link_name and state to monitor topic state distribution across your shadow links. | | redpanda_shadow_link_total_records_fetched | Count | The total number of records fetched by the sharded replicator (records received by the client). Monitor by shadow_link_name and shard to track message throughput from the source. | | redpanda_shadow_link_total_records_written | Count | The total number of records written by a sharded replicator (records written to the write_at_offset_stm). Uses shadow_link_name and shard labels to monitor message throughput to the shadow cluster. | See also: [Public Metrics](https://docs.redpanda.com/streaming/25.3/reference/public-metrics-reference/) ## [](#monitoring-best-practices)Monitoring best practices ### [](#health-check-procedures)Health check procedures Establish regular monitoring workflows to ensure shadow link health: #### Operator ```bash # Check all shadow links are synced and healthy kubectl get shadowlink --namespace # View detailed status for a specific shadow link kubectl describe shadowlink --namespace # Check for any shadow links with issues (not synced) kubectl get shadowlink --namespace -o json | \ jq '.items[] | select(.status.conditions[] | select(.type=="Synced" and .status!="True")) | .metadata.name' ``` #### Helm ```bash # Check all shadow links are active kubectl exec --namespace --container redpanda -- \ rpk shadow list | grep -v "ACTIVE" || echo "All shadow links healthy" # Monitor lag for critical topics kubectl exec --namespace --container redpanda -- \ rpk shadow status | grep -E "LAG|Lag" ``` ### [](#alert-conditions)Alert conditions Configure monitoring alerts for the following conditions, which indicate problems with Shadowing: - **High replication lag**: When `redpanda_shadow_link_shadow_lag` exceeds your RPO requirements - **Connection errors**: When `redpanda_shadow_link_client_errors` increases rapidly - **Topic state changes**: When topics move to `FAULTED` state - **Task failures**: When replication tasks enter `FAULTED` or `NOT_RUNNING` states - **Throughput drops**: When bytes/records fetched drops significantly - **Link unavailability**: When tasks show `LINK_UNAVAILABLE` indicating source cluster connectivity issues For more information about shadow link tasks and their states, see [Shadow link tasks](https://docs.redpanda.com/streaming/25.3/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks).