Collapse navigation tree

Collapse

Create a BYOVPC Cluster on AWS

BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or Redpanda Sales.

A Bring Your Own Virtual Private Cloud (BYOVPC) cluster allows you to deploy the Redpanda data plane into your existing VPC and manage the networking lifecycle yourself. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more control. For background on the architecture, see BYOC architecture.

When you create a BYOVPC cluster, you specify your VPC and the IAM role (instance profile) that the Redpanda agent will assume. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With BYOVPC:

  • You provide your own VPC in your AWS account.

  • You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters.

  • You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets.

The Redpanda BYOVPC Terraform Module contains Terraform code that deploys the resources required for a BYOVPC cluster on AWS. You need to create these resources in advance and provide them to Redpanda during cluster creation. Variables are provided in the code so you can exclude resources that already exist in your environment, such as the VPC.

Secrets management is enabled by default with the Terraform module. It allows you to store and read secrets in your cluster, for example to integrate a REST catalog with Iceberg-enabled topics.

For existing BYOVPC clusters, contact Redpanda Support to enable secrets management.

Prerequisites

  • Access to an AWS account in which you create your cluster.

  • Minimum permissions in that AWS account. For the actions required by the user who will create the cluster with terraform apply, see iam_rpk_user.tf.

  • Each BYOVPC cluster requires one allocated Elastic IP (EIP) address in AWS.

  • Terraform version 1.8.5 or later.

  • The Redpanda Terraform provider configured with valid credentials. For setup details, see the provider documentation.

Limitations

  • Existing clusters cannot be converted to BYOVPC clusters.

  • After creating a BYOVPC cluster, you cannot change to a different VPC.

  • Only primary CIDR ranges are supported for the VPC.

For simplicity, the instructions are based on the assumption that Terraform is configured to use local state. You may want to configure remote state.

Configure the Redpanda BYOVPC Terraform module

The following example uses the Redpanda BYOVPC Terraform Module to create the resources required to create a BYOVPC cluster.

Redpanda recommends using a VPC in AWS with a CIDR block (10.0.0.0/16) to allow for enough address space. The subnets must be set to /24. 
locals {
  common_prefix               = "abc-stg"
  region                      = "us-east-2"
  zones                       = ["use2-az1", "use2-az2", "use2-az3"]
  enable_private_link         = false # see example below for enabling private link
  force_destroy_cloud_storage = false

  # see example below if using pre-existing VPC and subnets,
  # otherwise when provided with these cidrs the module will
  # attempt to create the VPC and subnets
  vpc_cidr_block       = "10.0.0.0/16"
  public_subnet_cidrs  = [
    "10.0.1.0/24", "10.0.3.0/24", "10.0.5.0/24",
    "10.0.7.0/24", "10.0.9.0/24", "10.0.11.0/24"
  ]
  private_subnet_cidrs = [
    "10.0.0.0/24", "10.0.2.0/24", "10.0.4.0/24",
    "10.0.6.0/24", "10.0.8.0/24", "10.0.10.0/24"
  ]

  # condition_tags restrict the IAM permissions granted by the
  # module to only those resources with these tags, when using
  # condition_tags these tags must also be provided to the
  # redpanda_cluster so that all resources created are given
  # these tags
  condition_tags = { "redpanda-managed" : "true" }

  # default_tags are applied to all resources created by the
  # module or redpanda_cluster resource
  default_tags = { "env" : "staging" }

  # when using a brand new AWS account that has never hosted an
  # EKS cluster before the EKS node group service linked role
  # must be created, if it already exists this may be set to false
  create_eks_nodegroup_service_linked_role = true
}

module "redpanda_byovpc" {
  source                                   = "redpanda-data/redpanda-byovpc/aws"
  common_prefix                            = local.common_prefix
  region                                   = local.region
  zones                                    = local.zones
  create_rpk_user                          = false
  enable_private_link                      = local.enable_private_link
  force_destroy_cloud_storage              = local.force_destroy_cloud_storage
  enable_redpanda_connect                  = true
  vpc_cidr_block                           = local.vpc_cidr_block
  private_subnet_cidrs                     = local.private_subnet_cidrs
  public_subnet_cidrs                      = local.public_subnet_cidrs
  condition_tags                           = local.condition_tags
  default_tags                             = local.default_tags
  create_eks_nodegroup_service_linked_role = local.create_eks_nodegroup_service_linked_role
}

  • To send telemetry back to the Redpanda control plane, the cluster needs outbound internet access. You can provide this through at least one public subnet, or through network peering or a transit gateway to another VPC that routes traffic through a public subnet. The example configuration includes multiple public subnets to allow for future scaling.

  • The example creates an Internet Gateway and an associated Route Table rule that routes traffic into the VPC, which allows the Redpanda control plane to access the cluster. To disable creation of the Internet Gateway, either remove the configuration and value for create_internet_gateway or set "create_internet_gateway": false.

  • When using a pre-existing VPC, at least one public subnet must already exist in that VPC. Setting public_subnet_cidrs = [] only prevents the module from creating new ones.

See the full list of zones and tiers available with each provider in the Control Plane API reference.

Configure the Redpanda network and cluster

After provisioning the AWS infrastructure, configure the Redpanda network and cluster resources using the module outputs.

locals {
  resource_group_name = "staging"
  throughput_tier     = "tier-1-aws-v3-arm"
}

data "redpanda_resource_group" "staging" {
  name = local.resource_group_name
}

resource "redpanda_network" "network" {
  name              = "${local.common_prefix}-network"
  resource_group_id = data.redpanda_resource_group.staging.id
  cloud_provider    = "aws"
  region            = local.region
  cluster_type      = "byoc"
  customer_managed_resources = {
    aws = {
      management_bucket = {
        arn = module.redpanda_byovpc.management_bucket_arn
      }
      dynamodb_table = {
        arn = module.redpanda_byovpc.dynamodb_table_arn
      }
      vpc = {
        arn = module.redpanda_byovpc.vpc_arn
      }
      private_subnets = {
        arns = module.redpanda_byovpc.private_subnet_arns
      }
    }
  }
  depends_on = [module.redpanda_byovpc]
}

resource "redpanda_cluster" "cluster" {
  name              = "${local.common_prefix}-cluster"
  resource_group_id = data.redpanda_resource_group.staging.id
  cloud_provider    = "aws"
  region            = redpanda_network.network.region
  zones             = local.zones
  network_id        = redpanda_network.network.id
  cluster_type      = "byoc"
  connection_type   = "private"
  throughput_tier   = local.throughput_tier
  allow_deletion    = false
  tags = merge(local.condition_tags, local.default_tags)
  customer_managed_resources = {
    aws = {
      agent_instance_profile = {
        arn = module.redpanda_byovpc.agent_instance_profile_arn
      }
      cloud_storage_bucket = {
        arn = module.redpanda_byovpc.cloud_storage_bucket_arn
      }
      cluster_security_group = {
        arn = module.redpanda_byovpc.cluster_security_group_arn
      }
      connectors_node_group_instance_profile = {
        arn = module.redpanda_byovpc.connectors_node_group_instance_profile_arn
      }
      connectors_security_group = {
        arn = module.redpanda_byovpc.connectors_security_group_arn
      }
      k8s_cluster_role = {
        arn = module.redpanda_byovpc.k8s_cluster_role_arn
      }
      node_security_group = {
        arn = module.redpanda_byovpc.node_security_group_arn
      }
      permissions_boundary_policy = {
        arn = module.redpanda_byovpc.permissions_boundary_policy_arn
      }
      redpanda_agent_security_group = {
        arn = module.redpanda_byovpc.redpanda_agent_security_group_arn
      }
      redpanda_node_group_instance_profile = {
        arn = module.redpanda_byovpc.redpanda_node_group_instance_profile_arn
      }
      redpanda_node_group_security_group = {
        arn = module.redpanda_byovpc.redpanda_node_group_security_group_arn
      }
      utility_node_group_instance_profile = {
        arn = module.redpanda_byovpc.utility_node_group_instance_profile_arn
      }
      utility_security_group = {
        arn = module.redpanda_byovpc.utility_security_group_arn
      }
      redpanda_connect_node_group_instance_profile = {
        arn = module.redpanda_byovpc.redpanda_connect_node_group_instance_profile_arn
      }
      redpanda_connect_security_group = {
        arn = module.redpanda_byovpc.redpanda_connect_security_group_arn
      }
    }
  }
  depends_on = [redpanda_network.network]
}

Apply the Terraform configuration

Initialize, plan, and apply Terraform to set up the AWS infrastructure:

terraform init &&
terraform plan &&
terraform apply

Cluster provisioning can take up to 45 minutes. When provisioning completes, the cluster status updates to Running. If the cluster stays in Creating status, contact Redpanda Support.

Validation checks

The redpanda_cluster resource performs validation checks before proceeding with provisioning:

  • RPK user: Checks if the user running the command has sufficient privileges to provision the agent. Any missing permissions are displayed in the output.

  • IAM instance profile: Checks that agent_instance_profile, connectors_node_group_instance_profile, redpanda_node_group_instance_profile, redpanda_connect_node_group_instance_profile, utility_node_group_instance_profile, and k8s_cluster_role have the minimum required permissions. Any missing permissions are displayed in the output.

  • Storage: Checks that the management_bucket exists and is versioned, checks that the cloud_storage_bucket exists and is not versioned, and checks that the dynamodb_table exists.

  • Network: Checks that the VPC exists, checks that the subnets exist and have the expected tags, and checks that the security groups exist and have the desired ingress and egress rules.

Delete the cluster

To delete the cluster and all associated resources, run terraform destroy.

This also deletes the customer-managed resources created by the module. 
terraform destroy

PrivateLink can be enabled during cluster creation or on an already existing cluster.

Start by enabling PrivateLink in the Redpanda BYOVPC Terraform module. This adds the permissions required for PrivateLink.

module "redpanda_byovpc" {
  # ...
  enable_private_link = true
  # ...
}

Enable PrivateLink on the redpanda_cluster resource:

resource "redpanda_cluster" "cluster" {
  # ...
  aws_private_link = {
    allowed_principals = ["arn:aws:iam::${var.aws_account_id}:root"]
    enabled            = true
    connect_console    = false
  }
  # ...
}

Deploy with pre-existing VPC and subnets

If you already have a VPC and subnets in your AWS account, provide their IDs to the module instead of CIDR blocks.

module "redpanda_byovpc" {
  # ...
  # vpc_cidr_block       = local.vpc_cidr_block
  # private_subnet_cidrs = local.private_subnet_cidrs
  # public_subnet_cidrs  = local.public_subnet_cidrs
  vpc_id              = "vpc-0c79b236047faa1ab"
  private_subnet_ids  = [
    "subnet-0e58df59b5eb037c3",
    "subnet-0c74559ab372f5123",
    "subnet-0525df35c467cad1c",
    "subnet-09c301e004e96c803",
    "subnet-0f67e76738572cb8e",
    "subnet-0cca6892cf789f6ec",
  ]
  public_subnet_cidrs = [] # when empty the module will not create any public subnets
  # ...
}

Next steps