Streaming

Linux System Tuning

Redpanda includes several features to tune your Linux systems for optimal performance. You can trigger hardware-based optimizations through the rpk redpanda tune command, also called the autotuner. Software-based configurations are managed automatically through the systemd suite of tools.

User-triggered hardware tuning

You can invoke the Redpanda autotuner using the rpk redpanda tune command to optimize native Linux nodes. It analyzes the hardware configuration of the worker node and sets appropriate kernel options. You can choose to set certain hardware parameters using ulimit or other system tools while using the autotuner to manage others.

Using the autotuner

The Redpanda autotuner has a large array of hardware tuners to choose from. You may choose to either execute all available tuners, or to apply only a subset of them. This is particularly important when you wish to maintain specific configurations and prevent Redpanda from altering these settings.

Redpanda recommends running the autotuner as part of your production deployment workflows. This ensures any hardware configurations or updates to the autotuner itself are incorporated quickly and efficiently into your system.

For the full list of available tuners, refer to rpk redpanda tune. When using Kubernetes, you should tune your system at the host node level. For more information on Kubernetes configuration and execution, refer to Kubernetes worker node tuning.

Automated software configurations

Redpanda makes use of systemd to manage operating system and software-based limits, such as memlocks, file handle limits, core limits, and certain distribution-specific configurations. These help ensure the system executes stably and efficiently. Some key examples of automated configurations include:

  • Detection of hung services and infinite looping scenarios. This helps ensure you don’t have blocking services or processes that may interfere with Redpanda’s stream processing.

  • Configuration of asynchronous input/output interfaces and scheduler affinity. This is particularly critical when you rely on real-time processing of data streams for your applications.

  • Use of systemd slices (see this blog post for more details), which allows Redpanda to efficiently use system resources while isolating tasks for scalability and performance reasons.

Open file limits

Check these limits before you deploy, and whenever brokers log Too many open files errors or topic creation fails because of the file descriptor limit.

Each partition replica on a broker uses several file descriptors, and each client and internal RPC connection uses another. If the broker reaches its open file limit, it can’t open new files or accept new connections.

Three Linux limits apply to the Redpanda process:

Limit Description

fs.file-max

Sysctl that sets the maximum number of file handles the kernel allocates across all processes on the host. Its default depends on the kernel and the amount of memory on the host.

fs.nr_open

Sysctl that sets the maximum value of a per-process open file limit. You can’t raise a process’s hard limit above this value.

Per-process limit (RLIMIT_NOFILE)

The soft and hard limit on open files for the Redpanda process, which ulimit -n reports. The soft limit is the limit that Redpanda runs with.

rpk redpanda tune doesn’t change any of these limits, and Redpanda runs with the soft limit it inherits from the process that starts it.

Per-process limit set by the Redpanda packages

The redpanda systemd service installed by the Redpanda Linux packages sets LimitNOFILE=800000, which sets both the soft and hard limits to 800,000. This limit applies only when you start Redpanda through the redpanda service. If you start Redpanda another way, such as in a container or with a custom unit file, the limit comes from whatever starts the process.

To change the limit, create a systemd drop-in file for the service instead of editing the packaged unit file, which package upgrades replace:

  1. Open a drop-in file for the redpanda service:

    sudo systemctl edit redpanda

  2. Add the limit and save the file:

    [Service]
LimitNOFILE=<limit>

    A process can’t have a limit higher than fs.nr_open. If you set a higher value, systemd applies the value of fs.nr_open instead.

  3. Restart Redpanda on the broker for the change to take effect:

    sudo systemctl restart redpanda

To persist kernel limits across reboots, set them in a file under /etc/sysctl.d/. For example, create /etc/sysctl.d/99-redpanda.conf:

fs.file-max = <system-limit>
fs.nr_open = <process-limit-ceiling>

Then load the settings:

sudo sysctl --system

Check the current limits

To check the kernel limits on a host:

sysctl fs.file-max fs.nr_open

To check how many file handles are in use on the host, read /proc/sys/fs/file-nr. The first value is the number of allocated file handles, and the third value is fs.file-max:

cat /proc/sys/fs/file-nr

To check the soft and hard limits of the running Redpanda process:

grep 'Max open files' /proc/$(pidof redpanda)/limits

Redpanda also logs its limits at startup in the form File handle limit: <soft>/<hard>. For example, on a host that uses the redpanda systemd service:

journalctl -u redpanda | grep 'File handle limit'

Effect on partition creation

Redpanda uses the open file limit to cap the number of partition replicas in the cluster. When a request would create new partitions, such as creating a topic or adding partitions to a topic, Redpanda refuses the request if the cluster would exceed this file descriptor limit. For how Redpanda calculates this limit and the other partition limits, see Partition count.

Only the controller leader runs this check, and it uses its own soft open file limit for every broker. Set the same open file limit on every broker so that the check matches the limits that brokers actually run with. When Redpanda refuses the request, the controller leader logs a warning that begins with Refusing to create and ends with would exceed FD limit <value>. The controller leader might not be the broker that the client sent the request to, so check the logs of the controller leader. To find the controller leader, run rpk cluster health and look for the Controller ID field.

Monitoring configurations

While Redpanda provides numerous metrics, the configurations applied using the autotuner and systemd are not exposed internally. You should continue monitoring your system health through standard means. For example, if you are concerned about approaching the open file limit, monitor it through standard system monitoring, such as the lsof utility or /proc/sys/fs/file-nr.