Configure a Customer-Managed VPC on AWS

Beta

To unlock this feature for your account, contact Redpanda support.

With a standard BYOC cluster, Redpanda manages the VPC lifecycle. For additional security, you can deploy the Redpanda data plane into your existing shared VPC and manage the VPC lifecycle yourself. When you create a BYOC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With a customer-managed VPC:

  • You provide your own VPC in your AWS account.

  • You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters.

  • You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets.

Prerequisites

  • Familiarity with the Redpanda Cloud API. For example, you should familiarize yourself with how to use the Cloud API to authenticate and create a cluster.

  • Access to an AWS project in which you create your cluster.

  • Minimum permissions in that AWS project. For the actions required by the user who will create the cluster with rpk cloud byoc aws apply, see iam_rpk_user.tf.

Limitations

  • You cannot move existing clusters to a customer-managed VPC.

  • You cannot change to a different VPC after creating a cluster with a customer-managed VPC.

Create managed resources

For example Terraform code of the expected provisioned resources, see the customer-managed VPC on AWS README. The ARNs (Amazon Resource Names) output by this Terraform code are necessary in later steps.

Create a network with the Cloud API

After authenticating to the Cloud API, issue a create network request using the Terraform outputs.

For information about the fields, see Create Network.

curl -X POST "https://api.redpanda.com/v1beta2/networks" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer $YOUR_TOKEN" \
-d @- << EOF
{
	 "name":"<your network resource name>",
	 "resource_group_id": "<resource group id of the network>",
	 "cloud_provider":"CLOUD_PROVIDER_AWS",
	 "region":"<region>",
	 "cluster_type":"TYPE_BYOC",
	 "customer_managed_resources": {
	   "aws": {
	     "management_bucket": {
	       "arn": "<management_bucket_arn from terraform outputs>"
	     },
	     "dynamodb_table": {
	       "arn": "<dynamodb_table_arn from terraform outputs>"
	     },
	     "private_subnets": {
	       "arns": [<private_subnet_ids from terraform outputs>]
	     },
	     "vpc": {
	       "arn": "<vpc_arn from terraform outputs>"
	     }
	   }
	 }
}
EOF

The create network request returns a resource_id. For example:

{
   "operation":{
      "id":"cpas8k6r4up5li18auh0",
      "metadata":{
         "@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateNetworkMetadata",
         "network_id":"cpb338gekjj5i1cpj3t0"
      },
      "state":"STATE_IN_PROGRESS",
      "started_at":"2024-05-28T19:33:54.631Z",
      "type":"TYPE_CREATE_NETWORK",
      "resource_id":"cpb338gekjj5i1cpj3t0"
   }
}

Create a cluster with the Cloud API

To create a cluster, issue a create cluster request using the Terraform outputs.

For information about the fields, see Create Cluster.

curl -X POST "https://api.redpanda.com/v1beta2/clusters" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer $YOUR_TOKEN" \
-d @- << EOF
{
  "cloud_provider":"CLOUD_PROVIDER_AWS",
  "connection_type":"CONNECTION_TYPE_PRIVATE",
  "name":"<name of cluster>",
  "resource_group_id":"<resource group id of the network>",
  "network_id":"<resource_id of network from previous step>",
  "region":"<region>",
  "throughput_tier":"<throughput tier>",
  "type":"TYPE_BYOC",
  "zones":["<zone 1>", "<zone 2>", "<zone 3>"],
  "redpanda_version": "<redpanda version>",
  "customer_managed_resources": {
    "aws": {
      "agent_instance_profile": {
        "arn": "<agent_instance_profile_arn from terraform outputs>"
      },
      "connectors_node_group_instance_profile": {
        "arn": "<connectors_node_group_instance_profile_arn from terraform outputs>"
      },
      "redpanda_node_group_instance_profile": {
        "arn": "<redpanda_node_group_instance_profile_arn from terraform outputs>"
      },
      "utility_node_group_instance_profile": {
        "arn": "<utility_node_group_instance_profile_arn from terraform outputs>"
      },
      "connectors_security_group": {
        "arn": "<connectors_security_group_arn from terraform outputs>"
      },
      "node_security_group": {
        "arn": "<node_security_group_arn from terraform outputs>"
      },
      "utility_security_group": {
        "arn": "<utility_security_group_arn from terraform outputs>"
      },
      "redpanda_agent_security_group": {
        "arn": "<redpanda_agent_security_group_arn from terraform outputs>"
      },
      "redpanda_node_group_security_group": {
        "arn": "<redpanda_node_group_security_group_arn from terraform outputs>"
      },
      "cluster_security_group": {
        "arn": "<cluster_security_group_arn from terraform outputs>"
      },
      "k8s_cluster_role": {
        "arn": "<k8s_cluster_role_arn from terraform outputs>"
      },
      "cloud_storage_bucket": {
        "arn": "<cloud_storage_bucket_arn from terraform outputs>"
      },
      "permissions_boundary_policy": {
        "arn": "<permissions_boundary_policy_arn from terraform outputs>"
      }
    }
  },
  # <The following aws_private_link section is optional. For more information, see https://docs.redpanda.com/current/deploy/deployment-option/cloud/aws-privatelink/.>
  "aws_private_link": {
    "enabled": true,
    "allowed_principals": [<allowed principals>],
    "connect_console": <true|false>
  }
}
EOF

The create cluster request returns a resource_id, which is required in the next step. For example:

{
   "operation":{
      "id":"cpas8k6r4up5li18auhg",
      "metadata":{
         "@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateClusterMetadata",
         "cluster_id":"cpb33c8ekjj5i1cpj3v0"
      },
      "state":"STATE_IN_PROGRESS",
      "started_at":"2024-05-28T19:34:09.501Z",
      "type":"TYPE_CREATE_CLUSTER",
      "resource_id":"cpb33c8ekjj5i1cpj3v0"
   }
}

Create cluster resources

To create the initial cluster resources, run rpk cloud byoc aws apply. This creates an autoscaling group, an agent VM, and the following resources:

  • S3 objects

  • Launch template

  • Autoscaling group

You must have the iam_rpk_user.tf permissions described in the prerequisites. 
rpk cloud login \
  --save \
  --client-id='<client-id>’ \
  --client-secret='<client-secret>' \
  --no-profile

rpk cloud byoc aws apply \
  --redpanda-id='<resource_id of cluster from previous step>'

Output:

Checking RPK User... PASSED
Checking IAM Instance Profiles... PASSED
Checking Storage... PASSED
Checking Network... PASSED
Reconciling agent infrastructure...
Running apply	{"provisioner": "redpanda-bootstrap"}
Finished apply	{"provisioner": "redpanda-bootstrap"}
Running apply	{"provisioner": "redpanda-network"}
Finished apply	{"provisioner": "redpanda-network"}
Running apply	{"provisioner": "redpanda-agent"}
Finished apply	{"provisioner": "redpanda-agent"}
The Redpanda cluster is deploying. This can take up to 45 minutes. View status at https://cloud.redpanda.com/clusters/<resource_id of cluster from previous step>/overview.

The agent VM now is running and handles the remaining provisioning steps. This can take up to 45 minutes. When provisioning completes, the cluster status updates to Running. If the cluster remains in Creating status after 45 minutes, contact Redpanda support.

Validatation checks

The rpk cloud byoc aws apply command performs validation checks before proceeding with provisioning:

  • RPK user: Checks if the user running the command has sufficient privileges to provision the agent. Any missing permissions are displayed in the output.

  • IAM instance profile: Checks that connectors_node_group_instance_profile, redpanda_node_group_instance_profile, utility_node_group_instance_profile, and k8s_cluster_role have the minimum required permissions. Any missing permissions are displayed in the output.

  • Storage: Checks that the management_bucket exists and is versioned, checks that the cloud_storage_bucket exists and is not versioned, and checks that the dynamodb_table exists.

  • Network: Checks that the VPC exists, checks that the subnets exist and have the expected tags, and checks that the security groups exist and have the desired ingress and egress rules.

If you think validation errors are erroneous, you can rerun the command with the --no-validate tag to skip validation.

Check cluster status

You can check the cluster status with the Cloud API or the Redpanda Cloud UI.

Example using the returned operation_id:

curl -X GET "https://api.redpanda.com/v1beta2/operations/<operation_id of operation from previous step>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer $YOUR_TOKEN"

Example retrieving cluster:

curl -X GET "https://api.redpanda.com/v1beta2/clusters/<resource_id of cluster from previous step>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer $YOUR_TOKEN"