Scale Redpanda in Kubernetes

You can scale a cluster both vertically (by increasing or decreasing the resources available to existing brokers) and horizontally (by adding or removing brokers from the cluster).

Vertical scaling

Vertical scaling involves increasing or decreasing the amount of resources available to Redpanda brokers, referred to as scaling up or scaling down. Resources include hardware resources such as CPU cores, memory, and storage.

To scale a Redpanda cluster vertically, see Manage Pod Resources in Kubernetes.

If your existing worker nodes have either too many resources or not enough resources, you may need to move Redpanda brokers to new worker nodes that meet your resource requirements. This process involves:

  • Making sure the new worker nodes are available.

  • Deleting each worker node individually.

  • Deleting the Pod’s PersistentVolumeClaim (PVC).

  • Ensuring that the PersistentVolume’s (PV) reclaim policy is set to Retain to make sure that you can roll back to the original worker node without losing data.

For emergency scenarios in which a node unexpectedly fails or is decommissioned without warning, the PVCUnbinder can help protect your Redpanda data. For details, see Enable the PVCUnbinder.

Horizontal scaling

Horizontal scaling involves modifying the number of brokers in your cluster, either by adding new ones (scaling out) or removing existing ones (scaling in). In situations where the workload is variable, horizontal scaling allows for flexibility. You can scale out when demand is high and scale in when demand is low, optimizing resource usage and cost.

Redpanda does not support Kubernetes autoscalers. Autoscalers rely on CPU and memory metrics for scaling decisions, which do not fully capture the complexities involved in scaling Redpanda clusters. Always manually scale your Redpanda clusters as described in this topic.

Do not rely on Kubernetes autoscalers to scale your Redpanda brokers. Instead, prevent infrastructure-level autoscalers, such as Karpenter, from terminating nodes that host Redpanda Pods. For example, you can set the statefulset.podTemplate.annotations field in the Redpanda Helm values, or the statefulset.podTemplate.annotations field in the Redpanda custom resource to include:

karpenter.sh/do-not-disrupt: "true"

This annotation tells Karpenter not to disrupt the node on which the annotated Pod is running. This can help protect Redpanda brokers from unexpected shutdowns in environments that use Karpenter to manage infrastructure nodes.

Scale out

Scaling out involves adding more brokers to your Redpanda cluster. You may want to add more brokers to increase throughput, enhance high availability, and improve fault tolerance. Adding more brokers enables a more effective distribution of data across the cluster. This is particularly important when dealing with large datasets.

To add Redpanda brokers to your cluster:

  1. Ensure that you have one additional worker node for each Redpanda broker that you want to add. Each Redpanda broker requires its own dedicated worker node so that it has access to all resources. For more details, see Kubernetes Cluster Requirements and Recommendations.

  2. If you use local PersistentVolumes (PV), ensure that your additional worker nodes have local disks available that meet the requirements of the configured StorageClass. See Store the Redpanda Data Directory in PersistentVolumes.

  3. If you have external access enabled, make sure that your new node has the necessary node ports open to external clients. See Networking and Connectivity in Kubernetes.

  4. Verify that your cluster is in a healthy state:

    kubectl exec redpanda-0 --namespace <namespace> -- rpk cluster health

  5. Increase the number of replicas in the Helm values:

    • Operator

    • Helm

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    statefulset:
      replicas: <number-of-replicas>
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

    • --values

    • --set

    replicas.yaml
    statefulset:
  replicas: <number-of-replicas>
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values replicas.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set statefulset.replicas=<number-of-replicas>

  6. Wait until the StatefulSet is rolled out:

    kubectl --namespace <namespace> rollout status statefulset redpanda --watch

  7. Verify that your cluster is in a healthy state:

    kubectl exec redpanda-0 --namespace <namespace> -- rpk cluster health

Scale in

Scaling in is the process of removing brokers from your Redpanda cluster. You may want to remove brokers for cost reduction and resource optimization.

To scale in a Redpanda cluster, follow the instructions for decommissioning brokers in Kubernetes to safely remove brokers from the Redpanda cluster.