23.2 23.1 22.3 22.2 22.1 21.11

Enable Rack Awareness in Kubernetes

Rack awareness allows you to distribute replicas of the same partition across different racks to minimize data loss in the event of a rack failure. A rack is a failure zone that has one or more Redpanda brokers assigned to it.

When you create a topic, you specify the number of partitions for the topic and the number of partition replicas. By default, Redpanda determines where to place the replicas on the cluster such that each replica is on a different broker, if possible.

By defining different racks for a Redpanda cluster, you can specify a preference for the way partition replicas are assigned to brokers. When Redpanda places partition replicas, it takes into account whether a replica has already been placed on a broker in a particular rack. If so, Redpanda chooses a broker in a different rack. This way, partition replicas are distributed across different failure zones, which provides a measure of fault tolerance in the event that a broker or an entire rack becomes unavailable.

When rack awareness is enabled, Redpanda places replicas according to these criteria:

  • Number of racks vs. replicas - If the cluster has more racks than the number of replicas, each replica is placed on a broker in a unique rack. If the cluster has fewer racks than the number of replicas, some replicas are placed on brokers in the same rack.

  • Number of available CPU cores - Brokers with more available CPU cores are chosen over brokers with fewer available CPU cores.

  • Broker utilization - Brokers with fewer partitions are chosen over brokers with more partitions.

When you enable rack awareness in the Redpanda Helm chart, Kubernetes failure zones are treated as racks. Redpanda maps each rack to a failure zone and places partition replicas across them. For more details about Kubernetes failure zones, see the Kubernetes documentation.

Prerequisites

Make sure that each of your Node resources is annotated with a key/value pair that corresponds to a failure zone. The Helm chart assigns each Redpanda broker to a particular rack, according to the failure zone of the Kubernetes node on which the broker is running.

Annotate Node resources

In Kubernetes, failure zones are treated as racks. Redpanda maps each rack to a failure zone and places partition replicas across them. For more details about Kubernetes failure zones, see the Kubernetes documentation.

To assign a failure zone to your Kubernetes nodes, ensure that each of your Node resources is annotated with a key/value pair that corresponds to a failure zone. The Helm chart assigns each Redpanda broker to a particular rack, according to the failure zone of the Kubernetes node on which the broker is running.

Managed Kubernetes platforms usually annotate Node resources with the availability zone in which the node instance is hosted. For example topology.kubernetes.io/zone=us-east-1a.

To check the value of the topology.kubernetes.io/zone key, run the following:

kubectl get node \
-o=custom-columns=NODE:.metadata.name,ZONE:.metadata.annotations."topology\.kubernetes\.io/zone"

Example output:

NODE                 ZONE
example-worker       us-east-1a
example-worker2      us-east-1b
example-worker3      us-east-1c

If you don’t see any values in the Zone column, make sure to annotate your Node resources with key/value pairs that correspond to your fault-tolerance requirements. For example:

kubectl annotate node example-worker topology.kubernetes.io/zone=rack1
kubectl annotate node example-worker2 topology.kubernetes.io/zone=rack2
kubectl annotate node example-worker3 topology.kubernetes.io/zone=rack3

Configure rack awareness

To enable rack awareness in your Redpanda cluster, you need to override the default configuration in the Helm chart depending on the key you used to annotate Node resources with the availability zone.

topology.kubernetes.io/zone annotations

By default, the Helm chart sets the rack to the value in the well-known topology.kubernetes.io/zone key.

If your Node resources are annotated with the topology.kubernetes.io/zone key, do the following:

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    rackAwareness:
      enabled: true
    serviceAccount:
      create: true
    rbac:
      enabled: true
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

  • --values

  • --set

rack-awareness.yaml
rackAwareness:
  enabled: true
serviceAccount:
  create: true
rbac:
  enabled: true
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
--values rack-awareness.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set rackAwareness.enabled=true \
  --set serviceAccount.create=true \
  --set rbac.enabled=true
The serviceAccount and rbac configurations are required. These configurations allow the initialization container to securely read the node annotations using the Kubernetes API.

Other annotations

If your Node resources are annotated with a different key, set rackAwareness.nodeAnnotation to match your annotation key.

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    rackAwareness:
      enabled: true
      nodeAnnotation: 'example'
    serviceAccount:
      create: true
    rbac:
      enabled: true
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

  • --values

  • --set

rack-awareness.yaml
rackAwareness:
  enabled: true
  nodeAnnotation: 'example'
serviceAccount:
  create: true
rbac:
  enabled: true
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
--values rack-awareness.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set rackAwareness.enabled=true \
  --set rackAwareness.nodeAnnotation='example' \
  --set serviceAccount.create=true \
  --set rbac.enabled=true

Verify that rack awareness is enabled

After deploying Redpanda, make sure that rack awareness is enabled and configured on your Redpanda brokers.

  1. Make sure that your Node resources include the correct annotations. For example, to check the value of the topology.kubernetes.io/zone key, run the following:

    kubectl get node \
-o=custom-columns=NODE:.metadata.name,ZONE:.metadata.annotations."topology\.kubernetes\.io/zone"

    Example output:

    NODE                 ZONE
example-worker       us-east-1a
example-worker2      us-east-1b
example-worker3      us-east-1c

  2. Make sure that rack awareness has been enabled and configured on your Redpanda brokers:

    • TLS Enabled

    • TLS Disabled

    kubectl --namespace <namespace> exec -i -t redpanda-0 -c redpanda -- rpk cluster config get enable_rack_awareness -X admin.hosts=redpanda-0.redpanda.<namespace>.svc.cluster.local.:9644,redpanda-1.redpanda.<namespace>.svc.cluster.local.:9644,redpanda-2.redpanda.<namespace>.svc.cluster.local.:9644 -X admin.tls.ca=/etc/tls/certs/default/ca.crt -X admin.tls.enabled=true
    kubectl --namespace <namespace> exec -i -t redpanda-0 -c redpanda -- rpk cluster config get enable_rack_awareness -X admin.hosts=redpanda-0.redpanda.<namespace>.svc.cluster.local.:9644,redpanda-1.redpanda.<namespace>.svc.cluster.local.:9644,redpanda-2.redpanda.<namespace>.svc.cluster.local.:9644

    Example output:

    true

Next steps

Use rack awareness with Continuous Data Balancingto continually maintain the configured replication level, even after a rack failure. For a given partition, Redpanda tries to move excess replicas from racks that have more than one replica to racks that have no replicas.

Suggested reading