Linux System Tuning

Redpanda includes several features to tune your Linux systems for optimal performance. You can trigger hardware-based optimizations through the rpk redpanda tune command, also called the autotuner. Software-based configurations are managed automatically through the systemd suite of tools.

User-triggered hardware tuning

You can invoke the Redpanda autotuner using the rpk redpanda tune command to optimize native Linux nodes. It analyzes the hardware configuration of the worker node and sets appropriate kernel options. You can choose to set certain hardware parameters using ulimit or other system tools while using the autotuner to manage others.

Using the autotuner

The Redpanda autotuner has a large array of hardware tuners to choose from. You may choose to either execute all available tuners, or to apply only a subset of them. This is particularly important when you wish to maintain specific configurations and prevent Redpanda from altering these settings.

Redpanda recommends running the autotuner as part of your production deployment workflows. This ensures any hardware configurations or updates to the autotuner itself are incorporated quickly and efficiently into your system.

For the full list of available tuners, refer to rpk redpanda tune. When using Kubernetes, you should tune your system at the host node level. For more information on Kubernetes configuration and execution, refer to Kubernetes worker node tuning.

Automated software configurations

Redpanda makes use of systemd to manage operating system and software-based limits, such as memlocks, file handle limits, core limits, and certain distribution-specific configurations. These help ensure the system executes stably and efficiently. Some key examples of automated configurations include:

  • Detection of hung services and infinite looping scenarios. This helps ensure you don’t have blocking services or processes that may interfere with Redpanda’s stream processing.

  • Configuration of asynchronus input/output interfaces and scheduler affinity. This is particularly critical when you rely on real-time processing of data streams for your applications.

  • Use of systemd slices (see this blog post for more details), which allows Redpanda to efficiently use system resources while isolating tasks for scalability and performance reasons.

Monitoring configurations

While Redpanda provides numerous metrics, the configurations applied using the autotuner and systemd are not exposed internally. You should continue monitoring your system health through standard means. For example, the systemd configuration sets the file handle limit to 800,000 references. If you are concerned about approaching this limit, you should monitor it through standard system monitoring, such as the use of the lsof utility.