Automate Deployment for Production

You can deploy Redpanda for production on Linux with the Redpanda installation binary. If you use automation tools in your environment, like Terraform and Ansible, you can use those tools for your Redpanda production deployment.

Set up infrastructure with Terraform

1. Install Terraform. See the Terraform documentation.

2. To clone the deployment-automation repo:

   git clone https://github.com/redpanda-data/deployment-automation.git

3. To change into the directory:

   cd deployment-automation

4. Follow the specific instructions for AWS or GCP:

AWS

1. Within the deployment-automation folder, change into the aws directory:

   cd aws

2. Set AWS credentials. Terraform provides multiple ways to set the AWS secret and key. See the Terraform documentation.

3. To initialize Terraform:

   terraform init

4. To create the cluster:

   terraform apply

   Configuration options:

   Property	Description
   aws_region	The AWS region to use for deploying the infrastructure. Default: us-west-2
   nodes	The number of nodes to base the cluster on. Default: 3
   enable_monitoring	Creates a Prometheus/Grafana instance for monitoring the cluster. Default: true
   instance_type	The instance type on which Redpanda is deployed. Default: i3.8xlarge
   prometheus_instance_type	The instance type on which Prometheus and Grafana are deployed. Default: c5.2xlarge
   public_key_path	Path to the public key of the keypair used to access the nodes. Default: ~/.ssh/id_rsa.pub
   distro	Linux distribution to install. (This affects the distro_ variables.) Default: ubuntu-focal
   distro_ami	AWS AMI to use for each available distribution. These must be changed according to the chosen AWS region.
   distro_ssh_user	User used to ssh into the created EC2 instances.

   The following example creates a three-node cluster using i3.large instances.

   terraform apply -var="instance_type=i3.large"

GCP

1. Within the deployment-automation folder, change into the gcp directory:

   cd gcp

2. You need an existing subnet to deploy the VMs into. The subnet's attached firewall should allow inbound traffic on ports 22, 3000, 8082, 8888, 8889, 9090, 9092, 9644, and 33145. This module adds the rp-node tag to the deployed VMs, which can be used as the target tag for the firewall rule.

3. To initialize Terraform:

   terraform init

4. To create the cluster:

   terraform apply

   Configuration options:

   Property	Description
   region	The region to use for deploying the infrastructure. Default: us-west1
   zone	The region's zone to deploy the infrastructure. Default: a
   subnet	The name of an existing subnet to deploy the infrastructure.
   nodes	The number of nodes to base the cluster on. Keep in mind that one node is used as a monitoring node. Default: 1
   disks	The number of local disks to deploy on each machine. Default = 1
   image	The OS image running on the VMs. Default: ubuntu-os-cloud/ubuntu-1804-lts
   machine_type	The machine type. Default: n2-standard-2
   public_key_path	Path to the public key of the keypair used to access the nodes.
   ssh_user	The ssh user. Must match the one in the public ssh key's comments.

   The following example creates a three-node cluster using the subnet named redpanda-cluster-subnet.

   terraform apply -var nodes=3 -var subnet=redpanda-cluster-subnet -var public_key_path=~/.ssh/id_rsa.pub -var ssh_user=$USER

Next steps

Now you can install Redpanda either with the Redpanda installation binary or with Ansible.

Install Redpanda with Ansible

Whether you are using bare metal servers or Terraform, you can use Ansible to install Redpanda.

Prerequisites

Install Ansible. See the Ansible documentation.

Clone the GitHub repository

To clone the repo:

git clone git@github.com:redpanda-data/deployment-automation.git

To change into the directory:

cd deployment-automation

To install the required roles needed by Ansible:

ansible-galaxy install -r ansible/requirements.yml

Configure the hosts.ini file

The hosts.ini file is in the deployment-automation directory. If you used Terraform to deploy the instances, this file is updated automatically. If you did not use Terraform, you must update it manually. When you open the file, and you see something like the following:

[redpanda]
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=0
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1

[monitor]
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1

Under the [redpanda] section, replace the following:

Property	Description
ip	The public IP address of the machine.
ansible_user	The username for Ansible to use to ssh to the machine.
private_ip	The private IP address of the machine. This could be the same as the public IP address.
id	The node ID of the Redpanda instance. This must be unique for each host.

The [monitor] section is only relevant if you have Prometheus and Grafana installed on a given host. If you don't want to have this deployed, then remove the [monitor] section.

Run the Ansible playbook

To set up Redpanda on your selected nodes:

ansible-playbook --private-key <your_private_key> -i hosts.ini -v ansible/playbooks/provision-node.yml

After this completes, you have a fully running cluster.