24.1 beta 23.3 23.2 23.1

Automate Deployment for Production

You can deploy Redpanda for production on Linux with the Redpanda installation binary. If you use automation tools in your environment, like Terraform and Ansible, you can use those tools for your Redpanda production deployment.

Set up infrastructure with Terraform

  1. Install Terraform. See the Terraform documentation.

  2. To clone the deployment-automation repo:

    git clone https://github.com/redpanda-data/deployment-automation.git

  3. To change into the directory:

    cd deployment-automation

  4. Follow the specific instructions for AWS or GCP:

    • AWS

    • GCP

    1. Within the deployment-automation folder, change into the aws directory:

      cd aws

    2. Set AWS credentials. Terraform provides multiple ways to set the AWS secret and key. See the Terraform documentation.

    3. To initialize Terraform:

      terraform init

    4. To create the cluster:

      terraform apply

    Configuration options:

    Property Description

    aws_region

    The AWS region to use for deploying the infrastructure. Default: us-west-2

    nodes

    The number of nodes to base the cluster on. Default: 3

    enable_monitoring

    Creates a Prometheus/Grafana instance for monitoring the cluster. Default: true

    instance_type

    The instance type on which Redpanda is deployed. Default: i3.8xlarge

    prometheus_instance_type

    The instance type on which Prometheus and Grafana are deployed. Default: c5.2xlarge

    public_key_path

    Path to the public key of the keypair used to access the nodes. Default: ~/.ssh/id_rsa.pub

    distro

    Linux distribution to install. (This affects the distro_ variables.) Default: ubuntu-focal

    distro_ami

    AWS AMI to use for each available distribution. These must be changed according to the chosen AWS region.

    distro_ssh_user

    User used to ssh into the created EC2 instances.

    The following example creates a three-node cluster using i3.large instances.

    terraform apply -var="instance_type=i3.large"

    1. Within the deployment-automation folder, change into the gcp directory:

      cd gcp

    2. You need an existing subnet to deploy the VMs into. The subnet’s attached firewall should allow inbound traffic on ports 22, 3000, 8082, 8888, 8889, 9090, 9092, 9644, and 33145. This module adds the rp-node tag to the deployed VMs, which can be used as the target tag for the firewall rule.

    3. To initialize Terraform:

      terraform init

    4. To create the cluster:

      terraform apply
    Property Description

    region

    The region to use for deploying the infrastructure. Default: us-west1

    zone

    The region’s zone to deploy the infrastructure. Default: a

    subnet

    The name of an existing subnet to deploy the infrastructure.

    nodes

    The number of nodes to base the cluster on. Keep in mind that one node is used as a monitoring node. Default: 1

    disks

    The number of local disks to deploy on each machine. Default = 1

    image

    The OS image running on the VMs. Default: ubuntu-os-cloud/ubuntu-1804-lts

    machine_type

    The machine type. Default: n2-standard-2

    public_key_path

    Path to the public key of the keypair used to access the nodes.

    ssh_user

    The ssh user. Must match the one in the public ssh key’s comments.

    The following example creates a three-node cluster using the subnet named redpanda-cluster-subnet.

    terraform apply -var nodes=3 -var subnet=redpanda-cluster-subnet -var public_key_path=~/.ssh/id_rsa.pub -var ssh_user=$USER

Next steps

Now you can install Redpanda either with the Redpanda installation binary or with Ansible.

Install Redpanda with Ansible

Whether you are using bare metal servers or Terraform, you can use Ansible to install Redpanda.

Prerequisites

Install Ansible. See the Ansible documentation.

Clone the GitHub repository

To clone the repo:

git clone git@github.com:redpanda-data/deployment-automation.git

To change into the directory:

cd deployment-automation

To install the required roles needed by Ansible:

ansible-galaxy install -r ansible/requirements.yml

Configure the hosts.ini file

The hosts.ini file is in the deployment-automation directory. If you used Terraform to deploy the instances, this file is updated automatically. If you did not use Terraform, you must update it manually. When you open the file, and you see something like the following:

[redpanda]
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=0
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1

[monitor]
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1

Under the [redpanda] section, replace the following:

Property Description

ip

The public IP address of the machine.

ansible_user

The username for Ansible to use to ssh to the machine.

private_ip

The private IP address of the machine. This could be the same as the public IP address.

id

The node ID of the Redpanda instance. This must be unique for each host.

The [monitor] section is only relevant if you have Prometheus and Grafana installed on a given host. If you don’t want to have this deployed, then remove the [monitor] section.

Run the Ansible playbook

To set up Redpanda on your selected nodes:

ansible-playbook --private-key <your_private_key> -i hosts.ini -v ansible/playbooks/provision-node.yml

After this completes, you have a fully running cluster.