Source Cluster Fencing
During a failover, the source cluster of a shadow link can still be reachable, and clients can keep writing to it. Writes that the source cluster accepts after the shadow link stops replicating never reach the shadow cluster, so the clusters diverge, and you lose those writes. Fencing the source cluster before you fail over prevents this in two ways. First, the fenced cluster stops accepting writes, so the shadow link catches up to a fixed point instead of chasing new data. Second, while the fence stays up after the failover, the source cluster stays at that point, and every client must move to the shadow cluster.
After reading this page, you will be able to:
-
Configure the fencing settings so that a shadow link keeps replicating from a fenced cluster
-
Fence the source cluster before you fail over a shadow link
-
Release the fence when you decide not to fail over
Fence the source cluster for a planned cutover, a failover test, or a failover during a partial outage where the source cluster still answers. If the source cluster is unreachable, you can’t fence it. Fail over the shadow link without fencing. See Failover Runbook.
What source cluster fencing does
Source cluster fencing stops the source cluster from accepting new data and changes. A client that still points at the source cluster gets an error instead of writing data that the shadow cluster never receives. By default, the fence also blocks reads. A read policy lets you exempt principals so that they can keep reading while the cluster is fenced. For example, exempt the user that a shadow link authenticates as, so that the link keeps replicating.
Fencing doesn’t check whether a cluster is the source of a shadow link. A source cluster doesn’t know about the shadow links that replicate from it, so you can fence any cluster, including one that no shadow link replicates from.
The fence applies to user topics, consumer groups, Schema Registry, access control lists (ACLs), and roles. While the cluster is fenced, it rejects:
-
Produce requests to user topics.
-
Adding a partition or consumer group to a transaction.
-
Creating or deleting a topic, adding partitions to a topic, changing a topic’s configuration, or deleting a topic’s records.
-
Consumer group requests that change a group, such as JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, and OffsetDelete, and deleting a consumer group.
-
Schema Registry changes.
-
Creating or deleting ACLs.
-
Creating or deleting a role, and adding or removing role members. You can still read roles.
-
Fetch requests from user topics, unless the read policy exempts the client. See Read policy.
After brokers apply the fence, they abort open transactions in its scope instead of waiting for them to time out. Writes that a broker accepted before the fence can still finish, and partition offsets can still advance for a short time.
The fence survives broker restarts. A fenced cluster stays fenced until you disable fencing.
Read policy
By default, a fence blocks every consumer, including the shadow link. The shadow cluster fetches data from the source cluster, so a fence that blocks the shadow link stops replication. Before you fence the source cluster, choose a read policy:
-
Exempt principals: List the users who can still fetch from the fenced cluster, such as the user that your shadow link authenticates as. The fence blocks all other consumers. Use bare user names, such as
shadow-link-user, not theUser:shadow-link-userform. An empty list ([]) exempts no one, so the fence blocks fetches from every user. An empty name in the list ([""]) matches clients that connect without Simple Authentication and Security Layer (SASL) or mTLS. -
Allow all reads: Every consumer can keep fetching. The exempt principals list then has no effect.
A read policy allows only reads. The fence still rejects writes from every client, including exempt principals and clients that read under allow all reads. A consumer that writes back to the cluster as part of its work, for example by producing to another topic or by committing offsets in a consumer group, fails. Exempt only the users that need to read, such as the shadow link’s user, instead of allowing all reads as a shortcut.
If the fence blocks the shadow link’s user, replication stops. The shadow cluster calculates the lag from its own fetch requests to the source cluster. The lag then freezes at its last value instead of growing. That value can be 0, and the link can look like it has caught up. To spot a blocked link, watch redpanda_shadow_link_client_errors on the shadow cluster. It rises when the link’s fetches fail.
|
The fence saves these settings, and it uses them every time you enable fencing. Disabling fencing keeps the saved settings.
Fencing states
rpk cluster fencing status reports one of these states:
| State | Meaning |
|---|---|
|
Fencing is disabled. |
|
Some brokers haven’t applied the latest change yet. A broker that is down counts as a broker that hasn’t applied it. A settings change on a fenced cluster also puts it in this state until every broker has the new settings. |
|
Every broker enforces the fence, and keeps enforcing it after a restart. |
|
Some brokers haven’t stopped fencing yet. |
The status also reports a revision. The revision changes with each enable, disable, or settings change that changes something. It is -1 until the first change. The revision is an opaque value for detecting a change between commands. It doesn’t count changes, so it can jump, for example from 10 to 200, instead of rising by one. Pass it to --expected-revision so that a command fails if someone else changed fencing after you read the status.
Prerequisites
-
Redpanda 26.3 or later on the source cluster.
-
Superuser access to the source cluster. Every fencing operation, including the status check, requires a superuser.
-
rpk26.3 or later, with a profile that connects to the source cluster’s Admin API. -
If a shadow link replicates from the source cluster, the name of the user that the link authenticates as on the source cluster. Find it in
client_options.authentication_configurationin the shadow link configuration on the shadow cluster. See Configure Shadowing.
You can’t enable fencing in these cases:
-
The cluster is the shadow cluster of a shadow link that hasn’t failed over. This check applies only to shadow clusters.
-
A data migration is in progress. It must finish, or you must cancel or delete it.
-
A whole cluster restore is in progress.
Configure the read policy
Save the fencing settings before you enable fencing. If you don’t, the fence uses the default read policy: no exempt principals, and allow_all_reads set to false. The fence then blocks every consumer, including a shadow link. The config update command replaces all settings. A field that you leave out of the file goes back to its default.
-
Print a template of the settings to a file:
rpk cluster fencing config template > fencing.yaml
+ The template has the default settings and a description of each field.
-
Edit
fencing.yamland add the users who should keep read access to the cluster, such as the user that a shadow link authenticates as:
scope: all_resources
read_policy:
allow_all_reads: false
exempt_principals:
- <user-name>
+
all_resources is the only scope.
-
Apply the file:
rpk cluster fencing config update --config-file fencing.yaml
+ The command prints the revision of the saved settings:
+
Saved the fencing settings at revision <revision>.
+
If the settings block every consumer, or an exempt principal includes the User: prefix, the command prints a warning.
-
Check the saved settings:
rpk cluster fencing statusThe
EXEMPT PRINCIPALSrow lists the users that you added, andALLOW ALL READSisfalse.
Fence the source cluster and fail over
Run the fencing commands against the source cluster, and the shadow link commands against the shadow cluster.
-
On the source cluster, enable fencing and wait until every broker enforces it:
rpk cluster fencing enable --wait
+
The command shows the saved settings and asks you to confirm. In a script, add --no-confirm. While it waits, the command shows how many brokers confirmed the fence, for example STATE FENCING, REVISION <revision>, 2 of 3 brokers confirmed. When every broker enforces the fence, it prints:
+
Fencing is enabled.
+
If you press Ctrl+C, only the wait stops. Fencing continues on the cluster. If a broker is down, the state stays FENCING until the broker comes back. A broker that is down counts as a broker that hasn’t applied the fence.
-
On the source cluster, confirm that the state is
FENCEDand that every broker reports the same revision:rpk cluster fencing status -
On the shadow cluster, check the replication lag:
rpk shadow status <shadow-link-name>Writes that the source cluster accepted before the fence can still replicate. Wait until the lag reaches 0 before you fail over.
A lag of 0 alone doesn’t prove that the shadow link caught up. The lag freezes at its last value when the link’s fetches fail. Trust the lag only after you confirm two things. First, the user that the shadow link authenticates as is in the
EXEMPT PRINCIPALSrow ofrpk cluster fencing statuson the source cluster. Second,redpanda_shadow_link_client_errorson the shadow cluster stays flat.A network problem between the clusters can also freeze the lag. To be certain that the shadow cluster caught up, compare the high watermark of each partition on the two clusters. Skip read replica topics: the fence doesn’t block their writes, because a read replica topic gets its data from object storage, not through produce requests, so its partitions can keep growing while the fence is up. Run this command against the source cluster and then against the shadow cluster:
rpk topic describe <topic-name> --print-partitions
-
On the shadow cluster, fail over the shadow link. See Failover.
-
Point your clients at the shadow cluster, and confirm that they produce and consume there. See Update application configuration.
Keep the source cluster fenced after the failover. Releasing the fence lets any client that still points at the source cluster write to it again, and the clusters diverge.
| Don’t shadow read replica topics. Set up the read replica on the shadow cluster directly. |
Release the fence
Release the fence only if you decide not to fail over and want to keep using the source cluster. Don’t release it after a failover. A released source cluster accepts writes, changes, and fetches again, so any client that still points at it can write to it, and the clusters diverge.
-
On the source cluster, disable fencing and wait until every broker stops fencing:
rpk cluster fencing disable --waitWhen every broker has stopped fencing, the command prints:
Fencing is disabled. -
Confirm that the state is
UNFENCED:rpk cluster fencing status -
Check your clients. A client that didn’t handle the fence errors cleanly might not recover on its own. Restart any client that doesn’t produce or consume again.
The saved settings stay. The next rpk cluster fencing enable uses them.
Guard against concurrent changes
If more than one operator or automation manages fencing, pass the revision from rpk cluster fencing status to --expected-revision. The command fails if fencing changed after you read the status:
rpk cluster fencing enable --expected-revision <revision>
rpk cluster fencing disable and rpk cluster fencing config update take the same flag.
Fence through the Admin API
rpk cluster fencing calls the ClusterFenceService of the Admin API. To fence from your own automation, call the service directly. All calls require a superuser.
| Operation | rpk command |
|---|---|
|
|
|
|
|
|
|
|
|
|
Fence and Unfence return after the controller accepts the request. Poll GetClusterFenceStatus until it reports CLUSTER_FENCE_STATE_FENCED or CLUSTER_FENCE_STATE_UNFENCED. GetClusterFenceBrokerStatus answers even while the controller has no leader.
For example, this request fences the cluster. If Admin API authentication is enabled, pass a superuser’s credentials with -u:
curl -X POST http://<broker-address>:9644/redpanda.core.admin.v2.ClusterFenceService/Fence \
-u <superuser>:<password> \
-H 'Content-Type: application/json' \
-d '{}'
Monitor source cluster fencing
Redpanda exports these metrics on the /public_metrics endpoint:
| Metric | Description |
|---|---|
|
|
|
Time when the fence entered its current state, in seconds since the Unix epoch. |
|
Revision of the fence that this broker applied. A broker with a lower revision than the others hasn’t applied the latest change yet. |
|
Number of client requests that this broker refused because the cluster is fenced. The |
A rising redpanda_cluster_fence_refused_requests_total after you fail over to the shadow cluster means that some clients still point at the fenced source cluster.
What clients see when the fence rejects a request
Each API reports a request that the fence rejects in its own way.
Kafka API
The fence returns these Kafka error codes by default:
| Request that the fence rejects | Default error code |
|---|---|
Produce |
|
Adding a partition or consumer group to a transaction |
|
Creating or deleting a topic, adding partitions, changing topic configuration, or deleting records |
|
Consumer group requests, such as JoinGroup, Heartbeat, and OffsetCommit |
|
Fetch |
|
Creating or deleting ACLs |
|
After the fence refuses a request, Redpanda throttles the client’s next request of the same type, so that clients that retry right away don’t flood the fenced cluster. Clients handle the throttle automatically, so you don’t configure anything on the client.
Change the error codes
Cluster properties set the Kafka error codes and the throttle. You don’t need to change them. Change them only to test how your clients respond. A value of 0 isn’t accepted for an error code, because clients read it as success.
| Property | Request |
|---|---|
|
Produce |
|
Adding a partition or consumer group to a transaction |
|
Creating or deleting a topic, adding partitions, changing topic configuration, or deleting records |
|
Consumer group requests |
|
Fetch |
|
Creating or deleting ACLs |
cluster_fence_kafka_refusal_throttle_ms sets the throttle time after a refused request. The default is 500 milliseconds, and 0 turns off the throttle.