Deploying Redpanda for Production
This guide will take you through what is needed to setup a production cluster of Redpanda.
|Before you set up your Redpanda cluster, refer to the Performance and storage tuning documentation for guidelines on cluster creation, such as avoiding out-of-disk outages.|
For the best performance, we need to provision the hardware according to these hardware requirements:
XFS for the data directory of Redpanda (/var/lib/redpanda/data)
A kernel that is at least 3.10.0-514, but a 4.18 or newer kernel is preferred
Local NVMe, RAID-0 when using multiple disks
2GB of memory per core
33145- Internal RPC Port
9092- Kafka API Port
8082- Pandaproxy Port
8081- Schema Registry Port
9644- Prometheus and HTTP admin port
If you want, you can use Terraform to deploy Redpanda.
After the hardware is provisioned, install Redpanda and configure it for production use.
You can also install Redpanda using an Ansible playbook.
On Fedora/RedHat Systems:
curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | \ sudo -E bash && sudo yum install redpanda -y
On Debian Systems:
curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | \ sudo -E bash && sudo apt install redpanda -y
By default Redpanda is installed in development mode, which turns off hardware optimization. To enable hardware optimization, set Redpanda to run in production mode:
sudo rpk redpanda mode production
We then need to tune the hardware, which can be done by running the following on each node:
sudo rpk redpanda tune all
Optional: Benchmark your SSD
On taller machines it is recommended benchmarking your SSD. This can be done
If you are on AWS, GCP or Azure, creating a new instance and upgrading to an image with a recent Linux Kernel version is often the easiest way to work around bad devices.
Now that the software is installed we need to configure it. The first step is to setup the root node. The root node will start as a standalone node, and every other one will join it, forming a cluster along the way.
For the root node we’ll choose 0 as its ID.
--self tells the node which interface address to bind to. Usually you want that to be its private IP.
sudo rpk config bootstrap --id 0 --self <ip> && \ sudo systemctl start redpanda-tuner redpanda
For every other node, we just have to choose a unique integer id for it and let it know where to reach the root node.
sudo rpk config bootstrap --id <unique id> \ --self <private ip> \ --ips <root node ip> && \ sudo systemctl start redpanda-tuner redpanda
Working with schema registry article