Streaming

Tune Kubernetes Worker Nodes for Production

To get the best performance from your hardware, run the autotuner tool on each worker node that hosts a Redpanda broker. The autotuner identifies the hardware configuration of your worker node and optimizes the Linux kernel to give you the best performance.

You can run the autotuner in one of two ways:

From the Redpanda Operator or Helm chart (recommended): a privileged init container on each broker Pod runs the autotuner against the worker node every time the Pod starts. No access to the worker nodes is required, and the tuners are reapplied after a node restarts. Requires Redpanda Operator or Redpanda Helm chart 26.2.1 or later.
Manually on each worker node: install Redpanda on each worker node and run the autotuner yourself before deploying Redpanda. Use this option when your security policy doesn’t allow privileged containers with hostPath volumes, when your worker node images aren’t supported by the init container, or when you run an Operator or Helm chart version earlier than 26.2.1.

Whichever option you choose, you can also generate an I/O configuration file for your worker nodes.

Tune worker nodes with the Redpanda Operator or Helm chart

Starting in version 26.2.1, the Redpanda Operator and the Redpanda Helm chart can run the autotuner for you. When you set tuning.apply_host_tuners to true, each broker Pod gets a privileged tuning init container that runs rpk redpanda tune all against the host’s /sys, /proc, network interfaces, and block devices. The init container runs every time a broker Pod starts, so the tuners are reapplied after a worker node restarts.

Enabling apply_host_tuners turns on these tuners in addition to the chart defaults: tune_disk_irq, tune_disk_scheduler, tune_disk_nomerges, tune_network, tune_fstrim, tune_disk_write_cache (GCP only), and tune_cpu. It requires tune_aio_events: true, which is the default.

The tuning init container runs as a privileged container and mounts host paths such as /sys, /proc, /etc, and /dev. The namespace must allow the privileged Pod Security level or, on OpenShift, a security context constraint that allows hostPath volumes and privileged containers.

Run at most one Redpanda broker Pod on each worker node, which is the default Pod anti-affinity in the Helm chart. Tuners that run concurrently on the same worker node conflict with each other.

The tuning init container runs the autotuner from a chroot of the worker node’s filesystem. If a worker node doesn’t have /bin/bash or one of the required host directories, the broker Pod fails to start. If an individual tuner fails, the init container logs a warning and the broker starts with that tuner not applied.

Operator
Helm

redpanda-cluster.yaml

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tuning:
      tune_aio_events: true
      apply_host_tuners: true

values.yaml

tuning:
  tune_aio_events: true
  apply_host_tuners: true

Turning off apply_host_tuners or uninstalling Redpanda stops future tuning runs but does not undo tuning that was already applied. Kernel, IRQ, and block device settings persist until the worker node reboots, and the fstrim systemd timer stays active until you remove it from the host.

Disable individual tuners

To keep host tuning on but skip specific tuners, set the tuner’s rpk flag to false under config.rpk. Tuners that you don’t list keep their default of true. You can disable any of the tuners that apply_host_tuners enables: tune_disk_irq, tune_disk_scheduler, tune_disk_nomerges, tune_network, tune_fstrim, tune_disk_write_cache, and tune_cpu.

Redpanda Operator and Redpanda Helm chart versions 26.2.1 through 26.2.4 ignore explicit false values under config.rpk when apply_host_tuners is enabled, so every default tuner runs. To opt out of individual tuners, upgrade to version 26.2.5 or later.

Add the tuners to disable:

Operator
Helm

redpanda-cluster.yaml

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tuning:
      tune_aio_events: true
      apply_host_tuners: true
    config:
      rpk:
        tune_cpu: false
        tune_fstrim: false

values.yaml

tuning:
  tune_aio_events: true
  apply_host_tuners: true
config:
  rpk:
    tune_cpu: false
    tune_fstrim: false

Apply the change with helm upgrade or kubectl apply. The flags are written to redpanda.yaml, which is part of the Pod checksum, so the brokers roll and the tuning init container on each Pod reruns rpk redpanda tune all with the new flags. You don’t need to run any rpk commands on the worker nodes.
Verify the result in the init container log. Disabled tuners show false in the ENABLED column. The tune_network tuner is listed as net:
```
kubectl logs <broker-pod> --namespace <namespace> -c tuning | grep -E '^(TUNER|cpu|fstrim) '
```

Disabling a tuner does not undo changes it already made. Settings from a disabled tuner persist until the worker node reboots, and the fstrim systemd timer stays active until you remove it from the host.

For the full list of tuners and their rpk flags, see rpk redpanda tune list. For the apply_host_tuners field reference, see Redpanda CRD reference.

Tune worker nodes manually

If you can’t use the Redpanda Operator or Helm chart to tune your worker nodes, install Redpanda on each worker node that will host a broker and run the autotuner yourself.

Prerequisites

Make sure that your current Linux user has root privileges. The autotuner requires privileged access to the Linux kernel settings.

Install Redpanda

To run the autotuner or the I/O benchmark, you need to install the Redpanda binary on the worker node.

Fedora/RedHat
Debian/Ubuntu

# Run the setup script to download and install the repo
curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | sudo -E bash && \
# Use yum to install redpanda
sudo yum install redpanda -y

# Run the setup script to download and install the repo
curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | sudo -E bash && \
# Use apt to install redpanda
sudo apt install redpanda -y

Run the autotuner

To tune the Linux kernel of your worker nodes, run the autotuner on each worker node that will host a Redpanda broker.

Set Redpanda to production mode:
```
sudo rpk redpanda mode production
```

Run the autotuner:

sudo rpk redpanda tune all

Expected output:

TUNER                  APPLIED  ENABLED  SUPPORTED  ERROR
aio_events             true     true     true
ballast_file           true     true     true
clocksource            true     true     true
coredump               false    false    true
cpu                    true     true     true
disk_irq               true     true     true
disk_nomerges          true     true     true
disk_scheduler         true     true     true
disk_write_cache       false    true     false      Disk write cache tuner is only supported in GCP
fstrim                 false    false    true
net                    true     true     true
swappiness             true     true     true
transparent_hugepages  false    false    true

Changes to the Linux kernel are not persisted. If a worker node restarts, make sure to run sudo rpk redpanda tune all on it again.

You can use a privileged DaemonSet to schedule the autotuner on each worker node that runs a Redpanda broker. Apply taints to Nodes that successfully complete the autotuner command. Use tolerations on your Pods so that they are scheduled only on tuned worker nodes.

For details about the autotuner, including how to enable or disable an individual tuner, see the rpk redpanda tune command reference.

Check the open file limit

Redpanda runs with the open file limit that the container runtime on the worker node gives to containers. The Redpanda Helm chart and the Redpanda Operator do not set this limit, and the Kubernetes Pod spec has no field for it. A low limit lowers the number of partitions that the cluster accepts. For details about the limits involved, see Open file limits.

Container runtime defaults can change when you upgrade worker nodes. For example, the systemd unit file for containerd 2.0 no longer sets LimitNOFILE, so containers inherit the default limits of the host’s systemd. On GKE, node pools on version 1.33 use containerd 2.0, and the default soft limit for containers dropped to 1024. GKE fixed this in version 1.33.2-gke.1043000. For details, see the GKE known issues.

To check the soft and hard limits inside a Redpanda container:

kubectl exec <pod-name> --namespace <namespace> -c redpanda -- bash -c 'ulimit -S -n; ulimit -H -n'

Redpanda also logs its limits at startup in the form File handle limit: <soft>/<hard>:

kubectl logs <pod-name> --namespace <namespace> -c redpanda | grep 'File handle limit'

This log line is available only while the startup log is retained. If the startup log has rotated, use the ulimit command above.

If the limit is too low, raise it in the container runtime configuration of your worker nodes. Follow the documentation for your Kubernetes platform and container runtime.

Generate optimal I/O configuration settings

After tuning the Linux kernel, you can optimize Redpanda for the I/O capabilities of your worker node by using rpk to run benchmarks that capture its read/write IOPS and bandwidth capabilities. After running the benchmarks rpk saves the results to an I/O configuration file (io-config.yaml) that Redpanda can read upon startup to optimize itself for the worker node.

This step runs rpk on a worker node, so you need to install Redpanda on at least one worker node for each hardware type, even if you tune with the Redpanda Operator or Helm chart.

Unlike the autotuner, it isn’t necessary to run rpk iotune each time Redpanda is started, as its I/O output configuration file can be reused for each worker node that runs on the same type of hardware.

Run the I/O benchmark on your worker node:
```
rpk iotune
```
Example output:
/etc/redpanda/io-config.yaml
```
disks:
- mountpoint: /
  read_iops: 40952
  read_bandwidth: 5638210048
  write_iops: 6685
  write_bandwidth: 1491679488
```
When this command is successful, the I/O configuration file is saved to /etc/redpanda/io-config.yaml by default.
Copy the file to your local machine.
Create a ConfigMap in the same namespace in which you will deploy Redpanda to store the I/O configuration file:
```
kubectl create configmap redpanda-io-config --namespace <namespace> --from-file=<path-to-io-config-file>
```

You will mount this file onto the Pods that run Redpanda so that Redpanda can read it at startup. See Deploy Redpanda for Production in Kubernetes.

For more details about this procedure, see Optimize I/O.

Next steps

Deploy the Redpanda cluster.

Suggested labs

Search all labs

Was this helpful?

group Ask in the community

mail Share your feedback

What do you think of this page?

Let us know more:

Let us contact you about your feedback:

Tune Kubernetes Worker Nodes for Production

Tune worker nodes with the Redpanda Operator or Helm chart

Disable individual tuners

Tune worker nodes manually

Prerequisites

Install Redpanda

Run the autotuner

Check the open file limit

Generate optimal I/O configuration settings

Next steps

Suggested labs