Create a BYOVPC Cluster on GCP
|
BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or Redpanda Sales. |
A Bring Your Own Virtual Private Cloud (BYOVPC) cluster allows you to deploy the Redpanda data plane into your existing VPC and manage the networking lifecycle. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more control. To compare the two variants before you commit to one, see Choose Between BYOC and BYOVPC. For background on the architecture, see BYOC architecture.
When you create a BYOVPC cluster, you specify your VPC and service accounts. You create the networking and security resources in advance, so the Redpanda Cloud agent doesn’t create or change them. The agent still deploys and operates the Redpanda cluster infrastructure, using the service accounts you supply and only the permissions you grant them. With BYOVPC:
-
You provide your own VPC in your Google Cloud project.
-
You maintain more control of your Google Cloud project, because Redpanda requires fewer permissions than standard BYOC clusters.
-
You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets.
The Redpanda BYOVPC Terraform Module contains Terraform code that deploys the resources required for a BYOVPC cluster on GCP. The Redpanda Terraform provider then creates the Redpanda network and cluster from the module outputs, and runs the Redpanda agent bootstrap for you. You can do the whole deployment in a single Terraform configuration, with no gcloud or rpk commands. For every module input, see Redpanda BYOVPC Module Options for GCP.
If your clients need to connect from different GCP regions than where your cluster will be deployed, you must enable global access during cluster creation. See Enable Global Access.
After reading this page, you will be able to:
-
Deploy a BYOVPC cluster on GCP using the Redpanda BYOVPC Terraform module
-
Configure the Redpanda network and cluster resources using module outputs
-
Delete a BYOVPC cluster and the GCP resources that the module created
Prerequisites
-
A Google Cloud project in which you create your cluster. A standalone project is recommended. To use a shared VPC in a separate host project, see Use a shared VPC.
-
Permission in the project to create the resources that the module creates, including enabling APIs, creating service accounts and custom IAM roles, and setting IAM policy bindings. For example, the Owner role. The same identity creates the cluster, so it also needs the following permissions, which the Owner role includes:
-
compute.disks.create -
compute.disks.setLabels -
compute.instanceGroupManagers.create -
compute.instanceGroupManagers.delete -
compute.instanceGroupManagers.get -
compute.instanceGroups.create -
compute.instanceGroups.delete -
compute.instanceTemplates.create -
compute.instanceTemplates.delete -
compute.instanceTemplates.get -
compute.instanceTemplates.useReadOnly -
compute.instances.create -
compute.instances.setLabels -
compute.instances.setMetadata -
compute.instances.setTags -
compute.subnetworks.get -
compute.subnetworks.use -
compute.zones.list -
iam.roles.get -
iam.serviceAccounts.actAs -
iam.serviceAccounts.get -
resourcemanager.projects.get -
resourcemanager.projects.getIamPolicy -
serviceusage.services.list -
storage.buckets.get -
storage.buckets.getIamPolicy -
storage.objects.create -
storage.objects.delete -
storage.objects.get -
storage.objects.list
-
Google Cloud credentials for that identity. Terraform and the Redpanda agent bootstrap use Application Default Credentials, for example from
gcloud auth application-default login. To use a service account key instead, set theGOOGLE_CREDENTIALSenvironment variable to the contents of the key file, not its path:export GOOGLE_CREDENTIALS="$(cat <service-account-key-file>.json)" -
Terraform version 1.8.5 or later.
-
The Redpanda Terraform provider version 2.7.0 or later, configured with valid credentials. For example, set the
REDPANDA_CLIENT_IDandREDPANDA_CLIENT_SECRETenvironment variables. For setup details, see Redpanda Terraform Provider.
GCP quotas
Ensure at least three nodes of headroom in the relevant GCP quotas in the same region as your cluster. During maintenance, Redpanda may temporarily create extra nodes. Quotas such as vCPUs per VM family (for example, N2D) and Local SSD total per VM family (quota key: LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY) are listed for each tier on the Create BYOC cluster page in the Redpanda Cloud UI.
Headroom formulas:
-
vCPU spare =
3 x (vCPUs per node) -
Local SSD spare (GB) =
3 x (Storage size per node in GB)
For example, with per-node storage 1500 GB (4 × 375 GB Local SSD) and machine type n2d-standard-4 (4 vCPUs), keep 4500 GB Local SSD and 12 vCPUs of spare quota.
Limitations
-
Existing clusters cannot be converted to BYOVPC clusters.
-
After creating a BYOVPC cluster, you cannot change to a different VPC.
-
Only primary CIDR ranges are supported for the VPC.
-
Changing a required attribute under
customer_managed_resources.gcpplans a replacement of the network or cluster. See Configuration changes that replace a cluster.
|
For simplicity, these instructions assume that Terraform uses local state. You might want to configure remote state. |
Configure the Terraform providers
Create a directory for the Terraform configuration. In that directory, create a main.tf file that declares and configures the Redpanda and Google Cloud providers:
terraform {
required_version = ">= 1.8.5"
required_providers {
redpanda = {
source = "redpanda-data/redpanda"
version = ">= 2.7.0"
}
google = {
source = "hashicorp/google"
version = "~> 8.0"
}
}
}
provider "google" {
project = local.service_project_id
region = local.region
}
# Reads REDPANDA_CLIENT_ID and REDPANDA_CLIENT_SECRET from the environment.
provider "redpanda" {
# The Redpanda provider needs the project ID to run the agent bootstrap.
gcp_project_id = local.service_project_id
}
Configure the Redpanda BYOVPC Terraform module
Add the following to main.tf. This example uses the Redpanda BYOVPC Terraform module to create a new VPC, the subnet, and all the other resources required for a BYOVPC cluster.
locals {
# Project where Redpanda deploys the cluster.
service_project_id = "<service-project-id>"
# Project that owns the VPC. Use the service project ID, unless the
# VPC is a shared VPC in a separate host project.
network_project_id = "<service-project-id>"
region = "us-central1"
zones = ["us-central1-a", "us-central1-b", "us-central1-c"]
# Suffix added to the name of every resource that the module creates.
# Bucket names are global, so choose a value that no other deployment uses.
unique_identifier = "abc-stg"
}
module "redpanda_byovpc" {
source = "redpanda-data/redpanda-byovpc/gcp"
version = "~> 1.0"
service_project_id = local.service_project_id
network_project_id = local.network_project_id
region = local.region
unique_identifier = local.unique_identifier
# Set to true to let terraform destroy delete the buckets when they
# contain objects. Not recommended for production.
force_destroy_mgmt_bucket = false
force_destroy_cloud_storage_bucket = false
}
By default, the module:
-
Enables the Google APIs that Redpanda requires in the project.
-
Creates a VPC named
redpanda-network, with-<unique-identifier>appended when you setunique_identifier, and a subnet in the range10.0.0.0/24, a Pods secondary range10.0.8.0/21(redpanda-pods), and a Services secondary range10.0.1.0/24(redpanda-services). The GKE control plane uses10.0.7.240/28(gke_master_ipv4_cidr_block). -
Creates a Cloud Router and Cloud NAT for outbound traffic, and the firewall rules that Redpanda requires.
-
Creates the management bucket and the Tiered Storage bucket.
-
Creates the service accounts, custom IAM roles, and Workload Identity bindings for the Redpanda agent, cluster, Redpanda Console, connectors, Redpanda Connect, the Redpanda Connect API, the Redpanda Operator, and GKE.
| For the zones and throughput tiers available on GCP, see the Control Plane API reference. |
Optional: Deploy with a pre-existing VPC
Skip this section to let the module create the VPC. To deploy into a VPC and subnet that already exist, set vpc_name and subnet_name on the module. The subnet must have secondary ranges for Pods and Services with the names in secondary_ipv4_range_pods_name and secondary_ipv4_range_services_name. The module still creates the Cloud Router, Cloud NAT, and firewall rules in that VPC.
module "redpanda_byovpc" {
# ...
vpc_name = "existing-vpc-name"
subnet_name = "existing-subnet-name"
secondary_ipv4_range_pods_name = "existing-pods-range-name"
secondary_ipv4_range_services_name = "existing-services-range-name"
# ...
}
Optional: Use a shared VPC
If the VPC belongs to a separate host project, set network_project_id to the host project ID. When network_project_id differs from service_project_id, the module also grants the service project’s GKE service agent the roles it needs in the host project.
By default, the module also attaches the service project to the host project (attach_shared_vpc = true). It grants the roles/container.hostServiceAgentUser and roles/compute.securityAdmin roles on the host project with authoritative bindings, which remove any other members of those roles. If your shared VPC is already set up, set attach_shared_vpc = false and grant these roles yourself. When the module creates the VPC in the host project, it also sets an authoritative IAM policy on the subnet, which replaces every other IAM binding on that subnet. For more information, see the Google shared VPC documentation.
Configure the Redpanda network and cluster
Add the Redpanda resource group, network, and cluster to main.tf. These resources read the names of the GCP resources from the module outputs.
locals {
# A Redpanda Cloud resource group, not a GCP resource.
redpanda_cloud_resource_group_name = "byovpc"
throughput_tier = "tier-1-gcp-v2-x86"
}
resource "redpanda_resource_group" "byovpc" {
name = local.redpanda_cloud_resource_group_name
}
resource "redpanda_network" "network" {
name = "${local.unique_identifier}-network"
resource_group_id = redpanda_resource_group.byovpc.id
cloud_provider = "gcp"
region = local.region
cluster_type = "byoc"
customer_managed_resources = {
gcp = {
network_name = module.redpanda_byovpc.network_name
network_project_id = module.redpanda_byovpc.network_project_id
management_bucket = {
name = module.redpanda_byovpc.management_bucket_name
}
}
}
depends_on = [module.redpanda_byovpc]
}
resource "redpanda_cluster" "cluster" {
name = "${local.unique_identifier}-cluster"
resource_group_id = redpanda_resource_group.byovpc.id
network_id = redpanda_network.network.id
cloud_provider = "gcp"
region = redpanda_network.network.region
zones = local.zones
cluster_type = "byoc"
connection_type = "private"
throughput_tier = local.throughput_tier
allow_deletion = false
customer_managed_resources = {
gcp = {
subnet = {
name = module.redpanda_byovpc.subnet_name
secondary_ipv4_range_pods = {
name = module.redpanda_byovpc.secondary_ipv4_range_pods_name
}
secondary_ipv4_range_services = {
name = module.redpanda_byovpc.secondary_ipv4_range_services_name
}
k8s_master_ipv4_range = module.redpanda_byovpc.k8s_master_ipv4_range
}
agent_service_account = {
email = module.redpanda_byovpc.agent_service_account_email
}
console_service_account = {
email = module.redpanda_byovpc.console_service_account_email
}
connector_service_account = {
email = module.redpanda_byovpc.connector_service_account_email
}
redpanda_cluster_service_account = {
email = module.redpanda_byovpc.redpanda_cluster_service_account_email
}
gke_service_account = {
email = module.redpanda_byovpc.gke_service_account_email
}
tiered_storage_bucket = {
name = module.redpanda_byovpc.tiered_storage_bucket_name
}
# Redpanda Connect runs only when both of its service accounts are set.
redpanda_connect_service_account = {
email = module.redpanda_byovpc.redpanda_connect_service_account_email
}
redpanda_connect_api_service_account = {
email = module.redpanda_byovpc.redpanda_connect_api_service_account_email
}
# Secrets in cluster properties need the Redpanda Operator service account.
redpanda_operator_service_account = {
email = module.redpanda_byovpc.redpanda_operator_service_account_email
}
}
}
depends_on = [redpanda_network.network]
}
To use a Redpanda resource group that already exists, replace the redpanda_resource_group resource with a redpanda_resource_group data source, and reference data.redpanda_resource_group.byovpc.id in the network and cluster.
The last three service accounts in customer_managed_resources.gcp are optional, and the provider accepts them in version 2.7.0 and later:
-
redpanda_connect_service_accountandredpanda_connect_api_service_accountturn on Redpanda Connect. Redpanda turns it on only when both are set. -
redpanda_operator_service_accountis required for secrets in cluster properties.
To add them to a cluster that you created with an earlier provider version, upgrade the provider, add the three blocks, and run terraform apply. Terraform updates the cluster in place.
If you remove both Redpanda Connect service accounts from the configuration, the next terraform apply turns Redpanda Connect off on the cluster, and terraform plan warns you first. If you remove only one of them, terraform plan fails.
Apply the Terraform configuration
Initialize, plan, and apply Terraform to create the GCP infrastructure, the Redpanda network, and the Redpanda cluster:
terraform init &&
terraform plan &&
terraform apply
Terraform creates the resources in order: first the GCP resources in the module, then the Redpanda network, and then the Redpanda cluster. When it creates the cluster, the Redpanda Terraform provider runs the Redpanda agent bootstrap in your project, using your Google Cloud credentials. You don’t need to run rpk cloud byoc gcp apply yourself.
When provisioning completes, the cluster status updates to Running. If the cluster stays in Creating status, contact Redpanda Support.
Delete the cluster
The example configuration protects the cluster and its buckets. To delete them:
-
Set
allow_deletion = trueon theredpanda_clusterresource. -
Set
force_destroy_mgmt_bucketandforce_destroy_cloud_storage_buckettotrueon the module. Without these settings,terraform destroyfails on any bucket that still contains objects. -
Run
terraform apply, and then runterraform destroy:terraform apply && terraform destroy
Terraform deletes the cluster, runs the Redpanda agent teardown, and then deletes the Redpanda network and the GCP resources.
| This also deletes the customer-managed resources that the module created, including the Tiered Storage bucket and the data in it. |
| Redpanda Cloud does not support customer access or modifications to any of the internal data plane resources. This restriction allows Redpanda Data to manage all configuration changes internally to ensure a 99.99% service level agreement (SLA) for BYOC clusters. |
Manage custom resource labels and network tags
Your organization might require custom resource labels and network tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage labels and network tags on your cluster’s Dataplane settings page in the Redpanda Cloud UI, or with the Control Plane API. Redpanda reserves some label keys. See Reserved label and network tag keys.
|
Do not add labels or network tags directly to the GCP node pools of a BYOC cluster, either in the GCP console or with commands such as The Dataplane settings page and the Control Plane API are the supported methods. With both, Redpanda applies labels directly to the Google Compute Engine (GCE) instances and disks in your cluster, and network tags directly to the instances. Because nothing is applied to the node pools, GKE does not replace any nodes. |
Use the Dataplane settings page
To manage labels and network tags in the Redpanda Cloud UI:
-
In the Redpanda Cloud UI, open your cluster, and click Dataplane settings.
-
Under Manage resource labels and network tags for your cluster, click Add, and enter a key and value for each label. Keys and values are case-sensitive.
To apply a network tag to the GCE instances in the cluster, use a key that starts with
gcp.network-tag.. For example, the keygcp.network-tag.web-serversapplies theweb-serversnetwork tag.You can add up to 10 entries on this page. To manage up to 16, use the Control Plane API.
-
Click Save.
Use the Control Plane API
The Control Plane API allows up to 16 custom resource labels and network tags in GCP.
Make sure you have:
-
The cluster ID. You can find this in the Redpanda Cloud UI, in the page header of the cluster overview.
-
A valid bearer token for the Control Plane API. For details, see Authenticate to the API.
Then complete the following steps:
-
To refresh agent permissions so the Redpanda agent can update labels and network tags, run:
export CLUSTER_ID="<cluster-id>" export PROJECT_ID="<gcp-project-id>" rpk cloud byoc gcp apply --redpanda-id="$CLUSTER_ID" --project-id="$PROJECT_ID"This step is required because label/tag management requires additional IAM permissions that may not have been granted during initial cluster creation:
-
compute.disks.get -
compute.disks.list -
compute.disks.setLabels -
compute.instances.setLabels
-
-
To update labels and network tags, invoke the Control Plane API.
First, set your authentication token:
export AUTH_TOKEN="<your-bearer-token>"The
PATCHcall sets the labels and network tags specified under"cloud_provider_tags". It replaces the existing labels and tags with the specified labels and tags. Include all desired labels and tags in the request. To remove a single entry, omit it from the map you send.cluster_patch_body=$(cat <<'JSON' { "cloud_provider_tags": { "environment": "production", "cost-center": "engineering", "gcp.network-tag.web-servers": "true", "gcp.network-tag.database-access": "true" } } JSON ) curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body"To remove all labels and network tags, send an empty
cloud_provider_tagsobject:cluster_patch_body='{"cloud_provider_tags": {}}' curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body"Labels that Redpanda manages, such as the Google Cloud Marketplace label, remain.
Reserved label and network tag keys
Redpanda rejects the following label keys when you create the cluster, when you add labels on the Dataplane settings page, and when you update labels with the Control Plane API:
-
redpanda-org -
redpanda-id -
network-id -
redpanda-client
If you enter a reserved key in the UI, the UI shows the error This field is already in use by Redpanda. Redpanda matches keys exactly. Label keys must start with a lowercase letter and can contain only lowercase letters, numbers, underscores, and dashes, except for the gcp.network-tag. prefix.
Redpanda also reserves goog-partner-solution for all organizations. See Google Cloud Marketplace consumption-tracking label.
When you update labels with the Control Plane API, it rejects the request only if the request adds one of the four reserved keys or changes the value Redpanda set for it. You can send the reserved labels back unchanged, for example from a GET response, or leave them out.
Also avoid the following keys:
-
Keys that start with
redpanda-. Redpanda uses this prefix for its own labels, such asredpanda-managed. The Redpanda agent ignores custom labels with this prefix on GCE instances and disks, and the Redpanda Terraform provider drops them from state. -
Keys that start with
goog-gke-orgoog-k8s-. GKE sets these labels on node instances, and the Redpanda agent ignores custom labels with these prefixes.
Redpanda treats keys that start with gcp.network-tag. as network tag requests for the GCE instances in the cluster, not as labels.
Google Cloud Marketplace consumption-tracking label
If your organization subscribes to Redpanda Cloud through a Google Cloud Marketplace private offer, Redpanda adds the following label to the GCP resources it creates for your BYOC clusters:
goog-partner-solution=isol_plb32_0014m00001udwjbqaq_hsnezwtawmwgng3icvr677juyhbusiap
Google Cloud Marketplace requires this label for partner consumption tracking. Google uses it to attribute the Google Cloud resources that run your cluster to the Redpanda Marketplace listing. The value identifies the Redpanda listing and contains no information about your organization or your data.
The label appears on the following resources:
-
The GKE cluster
-
GCE instances and disks
-
Cloud DNS managed zones
-
Kubernetes node labels on Redpanda-managed node pools
-
The Tiered Storage bucket, the Compute Engine address resource, and the agent VM, which get the label the next time you run
rpk cloud byoc gcp apply
How the label behaves:
-
Redpanda applies the label to existing resources in place. Adding it doesn’t recreate resources, replace nodes, or restart brokers. This includes node pools: Redpanda adds it as a Kubernetes node label, which GKE updates without replacing nodes, not as a GCE label on the node pool.
-
Redpanda manages the label, so you can’t set or change it yourself. The Control Plane API rejects any update request that adds
goog-partner-solutionor changes its value, with the errorcloud tags cannot contain the reserved key "goog-partner-solution". If you include the key when you create a cluster, Redpanda ignores your value. If you omit the key from aPATCHrequest, Redpanda keeps the label on your cluster. -
The label doesn’t replace or change your own labels. It counts toward GCP’s per-resource label limit.
-
To label GCE instances and disks, the Redpanda agent needs the IAM permissions listed in Use the Control Plane API. On a BYOC cluster created before Redpanda added those permissions, run
rpk cloud byoc gcp applyto grant them. On a BYOVPC cluster, make sure the agent’s custom role includes them. The role that the Redpanda BYOVPC Terraform module creates already includes them, and so does the role in Configure the service project.
| For BYOVPC clusters, custom labels are not applied to the customer-managed resources that are deployed by the customer. |