Tune Kubernetes Worker Nodes for Production

To get the best performance from your hardware, set Redpanda to production mode on each worker node and run the autotuner tool. The autotuner identifies the hardware configuration of your worker node and optimizes the Linux kernel to give you the best performance.

Prerequisites

Make sure that your current Linux user has root privileges. The autotuner requires privileged access to the Linux kernel settings.

Install Redpanda

To run the autotuner, you need to install the Redpanda binary on each worker node.

  • Fedora/RedHat

  • Debian/Ubuntu

# Run the setup script to download and install the repo
curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | sudo -E bash && \
# Use yum to install redpanda
sudo yum install redpanda -y
# Run the setup script to download and install the repo
curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash && \
# Use apt to install redpanda
sudo apt install redpanda -y

Run the autotuner

To tune the Linux kernel of your worker nodes, run the autotuner on each worker node that will host a Redpanda broker.

  1. Set Redpanda to production mode:

    sudo rpk redpanda mode production

  2. Run the autotuner:

    sudo rpk redpanda tune all
    Expected output: 
    TUNER                  APPLIED  ENABLED  SUPPORTED  ERROR
aio_events             true     true     true
ballast_file           true     true     true
clocksource            true     true     true
coredump               false    false    true
cpu                    true     true     true
disk_irq               true     true     true
disk_nomerges          true     true     true
disk_scheduler         true     true     true
disk_write_cache       false    true     false      Disk write cache tuner is only supported in GCP
fstrim                 false    false    true
net                    true     true     true
swappiness             true     true     true
transparent_hugepages  false    false    true

Changes to the Linux kernel are not persisted. If a worker node restarts, make sure to run sudo rpk redpanda tune all on it again.

You can use a privileged DaemonSet to schedule the autotuner on each worker node that runs a Redpanda broker. Apply taints to Nodes that successfully complete the autotuner command. Use tolerations on your Pods so that they are scheduled only on tuned worker nodes.

For details about the autotuner, including how to enable or disable an individual tuner, see the rpk redpanda tune command reference.

Generate optimal I/O configuration settings

After tuning the Linux kernel, you can optimize Redpanda for the I/O capabilities of your worker node by using rpk to run benchmarks that capture its read/write IOPS and bandwidth capabilities. After running the benchmarks rpk saves the results to an I/O configuration file (io-config.yaml) that Redpanda can read upon startup to optimize itself for the worker node.

Unlike the autotuner, it isn’t necessary to run rpk iotune each time Redpanda is started, as its I/O output configuration file can be reused for each worker node that runs on the same type of hardware.

  1. Run the I/O benchmark on your worker node:

    rpk iotune

    Example output:

    /etc/redpanda/io-config.yaml
    disks:
- mountpoint: /
  read_iops: 40952
  read_bandwidth: 5638210048
  write_iops: 6685
  write_bandwidth: 1491679488

    When this command is successful, the I/O configuration file is saved to /etc/redpanda/io-config.yaml by default.

  2. Copy the file to your local machine.

  3. Create a ConfigMap in the same namespace in which you will deploy Redpanda to store the I/O configuration file:

    kubectl create configmap redpanda-io-config --namespace <namespace> --from-file=<path-to-io-config-file>

You will mount this file onto the Pods that run Redpanda so that Redpanda can read it at startup. See Deploy Redpanda for Production in Kubernetes.

For more details about this procedure, see Optimize I/O.

Next steps

Deploy the Redpanda cluster.

Suggested labs

Search all labs