23.2 23.1 22.3 22.2 22.1 21.11

Using Terraform and Ansible to Deploy Redpanda

You can deploy Redpanda for production on bare metal with the Redpanda installation binary.

If you use automation tools in your environment, like Terraform and Ansible, you can use those tools for your Redpanda production deployment.

Set up infrastructure with Terraform

  1. Install Terraform. See the Terraform documentation.

  2. To clone the deployment-automation repo:

     git clone https://github.com/redpanda-data/deployment-automation.git

  3. To change into the directory:

     cd deployment-automation

  4. Follow the specific instructions for AWS or GCP:

    • AWS

    • GCP

    1. Within the deployment-automation folder, change into the aws directory:

       cd aws

    2. Set AWS credentials. Terraform provides multiple ways to set the AWS secret and key. See the Terraform documentation.

    3. To initialize Terraform:

      terraform init

    4. To create the cluster:

      terraform apply

      Configuration options:

      Property Description

      aws_region

      The AWS region to use for deploying the infrastructure. Default: us-west-2

      nodes

      The number of nodes to base the cluster on. Default: 3

      enable_monitoring

      Creates a Prometheus/Grafana instance for monitoring the cluster. Default: true

      instance_type

      The instance type on which Redpanda is deployed. Default: i3.8xlarge

      prometheus_instance_type

      The instance type on which Prometheus and Grafana are deployed. Default: c5.2xlarge

      public_key_path

      Path to the public key of the keypair used to access the nodes. Default: ~/.ssh/id_rsa.pub

      distro

      Linux distribution to install. (This affects the distro_ variables.) Default: ubuntu-focal

      distro_ami

      AWS AMI to use for each available distribution. These must be changed according to the chosen AWS region.

      distro_ssh_user

      User used to ssh into the created EC2 instances.

    The following example creates a three-node cluster using i3.large instances.

    terraform apply -var="instance_type=i3.large"

    1. Within the deployment-automation folder, change into the gcp directory:

       cd gcp

    2. You need an existing subnet to deploy the VMs into. The subnet’s attached firewall should allow inbound traffic on ports 22, 3000, 8082, 8888, 8889, 9090, 9092, 9644, and 33145. This module adds the rp-node tag to the deployed VMs, which can be used as the target tag for the firewall rule.

    3. To initialize Terraform:

      terraform init

    4. To create the cluster:

      terraform apply

      Configuration options:

      Property Description

      region

      The region to use for deploying the infrastructure. Default: us-west1

      zone

      The region’s zone to deploy the infrastructure. Default: a

      subnet

      The name of an existing subnet to deploy the infrastructure.

      nodes

      The number of nodes to base the cluster on. Keep in mind that one node is used as a monitoring node. Default: 1

      disks

      The number of local disks to deploy on each machine. Default = 1

      image

      The OS image running on the VMs. Default: ubuntu-os-cloud/ubuntu-1804-lts

      machine_type

      The machine type. Default: n2-standard-2

      public_key_path

      Path to the public key of the keypair used to access the nodes.

      ssh_user

      The ssh user. Must match the one in the public ssh key’s comments.

    The following example creates a three-node cluster using the subnet named redpanda-cluster-subnet.

    ```bash
terraform apply -var nodes=3 -var subnet=redpanda-cluster-subnet -var public_key_path=~/.ssh/id_rsa.pub -var ssh_user=$USER
```

Next steps

Now you can install Redpanda either with the Redpanda installation binary or with Ansible.

Install Redpanda with Ansible

Whether you are using bare metal servers or Terraform, you can use Ansible to install Redpanda.

Prerequisites

Install Ansible. See the Ansible documentation.

Clone the GitHub repository

To clone the repo:

git clone git@github.com:redpanda-data/deployment-automation.git

To change into the directory:

cd deployment-automation

To install the required roles needed by Ansible:

ansible-galaxy install -r ansible/requirements.yml

Configure the hosts.ini file

The hosts.ini file is in the deployment-automation directory. If you used Terraform to deploy the instances, this file is updated automatically. If you did not use Terraform, you must update it manually. When you open the file, and you see something like the following:

[redpanda]
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=0
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1

[monitor]
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1

Under the [redpanda] section, replace the following:

Property Description

ip

The public IP address of the machine.

ansible_user

The username for Ansible to use to ssh to the machine.

private_ip

The private IP address of the machine. This could be the same as the public IP address.

id

The node ID of the Redpanda instance. This must be unique for each host.

The [monitor] section is only relevant if you have Prometheus and Grafana installed on a given host. If you don’t want to have this deployed, then remove the [monitor] section.

Run the Ansible playbook

To set up Redpanda on your selected nodes:

ansible-playbook --private-key <your_private_key> -i hosts.ini -v ansible/playbooks/provision-node.yml

After this completes, you have a fully running cluster.