Skip to main content
Version: 22.3

Writing Custom Deployment Automation

Redpanda supports several deployment tools for setting up a cluster:

  • Ansible Playbook
  • Helm Chart
  • Kubernetes Operator
tip

Redpanda strongly recommends using one of these supported deployment tools. See Deploying for Production.

This page provides information for anyone creating their own automation for deploying Redpanda clusters; that is, anyone not using one of these supported tools.

Configure bootstrapping

Redpanda cluster configuration is written with the Admin API and the rpk cluster config CLIs.

In the special case where you want to provide configuration to Redpanda before it starts for the first time, you can write a .bootstrap.yaml file in the same directory as redpanda.yaml.

This file is only read on the first startup of the cluster. Any subsequent changes to .bootstrap.yaml are ignored, so changes to cluster configuration must be done with the Admin API.

The content format is a YAML dictionary of cluster configuration properties. For example, to initialize a cluster with Admin API authentication enabled and a single superuser, the .bootstrap.yaml file would contain the following:

admin_api_require_auth: true
superusers:
- alice

With this configuration, the Admin API is not accessible until you bootstrap a user account.

Bootstrap a user account

When using username/password authentication, it's helpful to be able to create one user before the cluster starts for the first time.

Do this by setting the RP_BOOTSTRAP_USER environment variable when starting Redpanda for the first time. The value has the format <username>:<password>. For example, you could set RP_BOOTSTRAP_USER to alice:letmein.

note

RP_BOOTSTRAP_USER only creates a user account. You must still set up authentication using cluster configuration.

Secure the Admin API

The Admin API is used to create SASL user accounts and ACLs, so it's important to think about how you secure it when creating a cluster.

  • No authentication, but listening only on 127.0.0.1: This may be appropriate if your Redpanda processes are running in an environment where only administrators can access the host.
  • mTLS authentication: You can generate client and server x509 certificates before starting Redpanda for the first time, refer to them in redpanda.yaml, and use the client certificate when accessing the Admin API.
  • Username/password authentication: Use the combination of admin_api_require_auth, superusers, and RP_BOOTSTRAP_USER to access the Admin API username/password authentication. You probably still want to enable TLS on the Admin API endpoint to protect credentials in flight.

Configure the seed servers

Seed servers help new nodes join a cluster by directing requests from newly-started nodes to an existing cluster. The seed_servers node configuration property controls how Redpanda finds its peers when initially forming a cluster. It is dependent on the empty_seed_starts_cluster node configuration property.

Starting in Redpanda version 22.3, you should explicitly set empty_seed_starts_cluster to false on every node, and every node in the cluster should have the same value set for seed_servers. With this set of configurations, Redpanda clusters form with these guidelines:

  • When a node starts and it is a seed server (its address is in the seed_servers list), it waits for all other seed servers to start up, and it forms a cluster with all seed servers as members.
  • When a node starts and it is not a seed server, it sends requests to the seed servers to join the cluster.

It is essential that all seed servers have identical values for the seed_servers list. Redpanda strongly recommends at least three seed nodes when forming a cluster. Each seed server decreases the likelihood of unintentionally forming a split brain cluster. To ensure nodes can always discover the cluster, at least one seed node should be available at all times.

By default, for backward compatibility, empty_seed_starts_cluster is set to true, and Redpanda clusters form with the guidelines used prior to version 22.3:

  • When a node starts with an empty seed_servers list, it creates a single node cluster with itself as the only member.
  • When a node starts with a non-empty seed_servers list, it sends requests to the nodes in that list to join the cluster.

You should never have more than one node with an empty seed_servers list, which would result in the creation of multiple clusters.

info

Redpanda expects its storage to be persistent, and it's an error to erase a node's drive and restart it. However, in some environments (like when migrating to a different node pool on Kubernetes), truly persistent storage is unavailable, and nodes may find their data volumes erased. For such environments, Redpanda recommends setting empty_seed_starts_cluster to false and designating a set of seed nodes such that they couldn't lose their storage simultaneously.

Configure node IDs

Redpanda automatically generates unique node IDs for each new node. This means that you don't need to include node IDs in configuration files or worry about policies on node_id re-use.

If you choose to assign node IDs, make sure to use a fresh node_id each time you add a node to the cluster.

caution

Never reuse node IDs, even for nodes that have been decommissioned and restarted empty. Doing so can result in an inconsistent state.

Upgrade considerations

Deployment automation should place each node into maintenance mode and wait for it to drain leaderships before restarting it with a newer version of Redpanda. For information about how to drive a rolling upgrade of a Redpanda cluster, see Node Maintenance Mode.

Rolling restarts preserve high availability and reduce risk during upgrades. Check the cluster's health after upgrading each node. Rolling back an upgrade to an earlier feature release is only supported until the last node has been updated, so it's important to identify any issues before that point.