Create a BYOVPC Cluster on Azure

To unlock this feature for your account, contact Redpanda Support.

This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda data plane into your existing virtual network (VNet) and take full control of managing the networking lifecycle. Compared to a standard Bring Your Own Cluster (BYOC) setup, where Redpanda manages the networking lifecycle for you, this option provides more security.

When you create a BYOVPC cluster, you specify your VNet and managed identities. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With a customer-managed VNet:

You provide your own VNet in your Azure account.
You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters.
You control your security resources and policies, including subnets, user-assigned identities, IAM roles and assignments, security groups, storage accounts, and key vaults.

The Redpanda Cloud Examples repository contains Terraform code that deploys the resources required for a BYOVPC cluster on Azure. You’ll need to create these resources in advance and give them to Redpanda during cluster creation. Variables are provided in the code so you can exclude resources that already exist in your environment, such as the VNet.

See the code for the complete list of resources required to create and deploy Redpanda cluster. Customer-managed resources can be broken down into the following groups:

Resource group resources
User-assigned identities
IAM roles and assignments
Network
Storage
Key vaults

Prerequisites

Access to an Azure subscription where you want to create your cluster
Knowledge of your internal VNet and subnet configuration
Permission to call the Redpanda Cloud API
Permission to create, modify, and delete the resources described by Terraform
Terraform version 1.8.5 or later

Limitations

Existing clusters cannot be moved to a BYOVPC cluster.
After creating a BYOVPC cluster, you cannot change to a different VNet.

Set environment variables

Set environment variables for the resource group, VNet name, and Azure region. For example:

export AZURE_RESOURCE_GROUP_NAME=sample-redpanda-rg
export AZURE_VNET_NAME="sample-vnet"
export AZURE_REGION=eastus

Create Azure resource group and VNet

Create a resource group to contain all resources, and then create a VNet with your address and subnet prefixes.

The following example uses the environment variables to create the sample-redpanda-rg resource group and the sample-vnet virtual network with an address space of 10.0.0.0/16 and a subnet named default with a smaller range of 10.0.0.0/24.
```
az group create --name ${AZURE_RESOURCE_GROUP_NAME} --location ${AZURE_REGION}

az network vnet create \
    --name ${AZURE_VNET_NAME} \
    --resource-group $AZURE_RESOURCE_GROUP_NAME \
    --location ${AZURE_REGION} \
    --address-prefix 10.0.0.0/16 \
    --subnet-name default \
    --subnet-prefixes 10.0.0.0/24
```

Set additional environment variables for Azure resources. For example:

export AZURE_SUBSCRIPTION_ID=
export AZURE_TENANT_ID=
export AZURE_ZONES=["eastus-az2"]
export AZURE_RESOURCE_PREFIX=sample-

Configure Terraform

For simplicity, these instructions assume that Terraform is configured to use local state. You may want to configure remote state.

Create a JSON file called byovnet.auto.tfvars.json inside the Terraform directory to configure variables for your specific needs:

Show script

cat > byovnet.auto.tfvars.json <<EOF
{
  "azure_subscription_id": "${AZURE_SUBSCRIPTION_ID}",
  "azure_tenant_id": "${AZURE_TENANT_ID}",
  "azure_use_cli": "true",
  "region": "${AZURE_REGION}",
  "zones": ${AZURE_ZONES},
  "resource_name_prefix": "${AZURE_RESOURCE_PREFIX}",
  "tags": {},
  "redpanda_resource_group_name": "${AZURE_RESOURCE_GROUP_NAME}",
  "redpanda_storage_resource_group_name": "${AZURE_RESOURCE_GROUP_NAME}",
  "redpanda_network_resource_group_name": "${AZURE_RESOURCE_GROUP_NAME}",
  "redpanda_iam_resource_group_name": "${AZURE_RESOURCE_GROUP_NAME}",
  "redpanda_agent_identity_name": "agent-uai",
  "redpanda_agent_role_name": "agent-role",
  "redpanda_cert_manager_identity_name": "cert-manager-uai",
  "redpanda_external_dns_identity_name": "external-dns-uai",
  "redpanda_cluster_identity_name": "cluster-uai",
  "aks_identity_name": "aks-uai",
  "redpanda_console_identity_name": "console-uai",
  "redpanda_console_role_name": "console-role",
  "kafka_connect_identity_name": "",
  "redpanda_management_storage_account_name": "rpmgmtsa",
  "redpanda_management_storage_container_name": "rpmgmtsc",
  "redpanda_tiered_storage_account_name": "tieredsa",
  "redpanda_tiered_storage_container_name": "tieredsc",
  "redpanda_management_key_vault_name": "redpanda-vault",
  "redpanda_console_key_vault_name": "rp-console-vault",
  "redpanda_private_link_role_name": "pl-role",
  "vnet_name": "${AZURE_VNET_NAME}",
  "vnet_addresses": [
    "10.0.0.0/20"
  ],
  "private_subnets": {
    "agent-private": {
      "cidr": "10.0.3.0/24",
      "name": "snet-agent-private"
    },
    "connect-pod": {
      "cidr": "10.0.10.0/24",
      "name": "snet-connect-pods"
    },
    "connect-vnet": {
      "cidr": "10.0.11.0/24",
      "name": "snet-connect-vnet"
    },
    "kafka-connect-pod": {
      "cidr": "10.0.12.0/24",
      "name": "snet-kafka-connect-pods"
    },
    "kafka-connect-vnet": {
      "cidr": "10.0.13.0/24",
      "name": "snet-kafka-connect-vnet"
    },
    "rp-0-pods": {
      "cidr": "10.0.4.0/24",
      "name": "snet-rp-0-pods"
    },
    "rp-0-vnet": {
      "cidr": "10.0.5.0/24",
      "name": "snet-rp-0-vnet"
    },
    "rp-1-pods": {
      "cidr": "10.0.6.0/24",
      "name": "snet-rp-1-pods"
    },
    "rp-1-vnet": {
      "cidr": "10.0.7.0/24",
      "name": "snet-rp-1-vnet"
    },
    "rp-2-pods": {
      "cidr": "10.0.8.0/24",
      "name": "snet-rp-2-pods"
    },
    "rp-2-vnet": {
      "cidr": "10.0.9.0/24",
      "name": "snet-rp-2-vnet"
    },
    "system-pod": {
      "cidr": "10.0.1.0/24",
      "name": "snet-system-pods"
    },
    "system-vnet": {
      "cidr": "10.0.2.0/24",
      "name": "snet-system-vnet"
    }
  },
  "egress_subnets": {
    "agent-public": {
      "cidr": "10.0.0.0/24",
      "name": "snet-agent-public"
    }
  },
  "reserved_subnet_cidrs": {
    "k8s-service": "10.0.15.0/24"
  },
  "redpanda_security_group_name": "redpanda-nsg"
}
EOF

Deploy Terraform

Initialize, plan, and apply Terraform to set up the Azure infrastructure:

terraform init
terraform plan
terraform apply

Note the output values that the terraform apply command displays. The output values are necessary in later steps. To continue with cluster creation, see Create Azure VNet Redpanda Cluster.

You can also get these values by running terraform output. For example:

Show output from terraform output

agent_private_subnet_name = "my-snet-agent-private"
agent_user_assigned_identity_name = "my-agent-uai"
aks_user_assigned_identity_name = "my-aks-uai"
cert_manager_user_assigned_identity_name = "my-cert-manager-uai"
cluster_user_assigned_identity_name = "my-cluster-uai"
console_key_vault_name = "my-consolevault"
console_user_assigned_identity_name = "my-console-uai"
egress_subnet_name = "my-snet-agent-public"
external_dns_user_assigned_identity_name = "my-external-dns-uai"
iam_resource_group_name = "my-iam-rg"
kafka_connect_pods_subnet_name = "my-snet-kafka-connect-pods"
kafka_connect_user_assigned_identity_name = "my-kafka-connect-uai"
kafka_connect_vnet_subnet_name = "my-snet-kafka-connect-vnet"
management_bucket_storage_account_name = "mymanagement"
management_bucket_storage_container_name = "mymanagement"
management_key_vault_name = "my-redpandavault"
network_resource_group_name = "my-network-rg"
......
redpanda_resource_group_name = "my-redpanda-rg"
redpanda_security_group_name = "my-redpanda-nsg"
......
rp_0_pods_subnet_name = "my-snet-rp-0-pods"
rp_0_vnet_subnet_name = "my-snet-rp-0-vnet"
rp_1_pods_subnet_name = "my-snet-rp-1-pods"
rp_1_vnet_subnet_name = "my-snet-rp-1-vnet"
rp_2_pods_subnet_name = "my-snet-rp-2-pods"
rp_2_vnet_subnet_name = "my-snet-rp-2-vnet"
rp_connect_pods_subnet_name = "my-snet-connect-pods"
rp_connect_vnet_subnet_name = "my-snet-connect-vnet"
......
storage_resource_group_name = "my-storage-rg"
system_pods_subnet_name = "my-snet-system-pods"
system_vnet_subnet_name = "my-snet-system-vnet"
tiered_storage_account_name = "mytieredstorage"
tiered_storage_container_name = "mytieredstorage"
......
vnet_name = "my-rp-vnet"

The Terraform Readme has more information about variable files:

vars.azure.tf: Contains the variables related to Azure credentials.
vars.condition.tf: Contains the conditional variables; for example, on whether to create NAT.
vars.customer_input.tf: Contains the variables needed to create a cluster with the Cloud API.
vars.iam.tf: Contains the variables related to IAM roles.
vars.misc.tf: Contains the variables, regions, zones, tags, and resource name prefix.

Authenticate with Redpanda Cloud

Get a bearer token from the Cloud API authentication endpoint using your Redpanda credentials:

export REDPANDA_CLIENT_ID=
export REDPANDA_CLIENT_SECRET=

export BEARER_TOKEN=$(curl --request POST \
--url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
--header 'content-type: application/x-www-form-urlencoded' \
--data grant_type=client_credentials \
--data client_id=${REDPANDA_CLIENT_ID} \
--data client_secret=${REDPANDA_CLIENT_SECRET} \
--data audience=cloudv2-production.redpanda.cloud | jq -r '.access_token')

To get the Redpanda authentication credentials, follow the Redpanda Cloud API Quickstart.

Create network

To create the Redpanda network:

Define a JSON file called redpanda-network.json to configure the network for Redpanda with details about VNet, subnets, and storage.

Show script

cat > redpanda-network.json <<EOF
{
  "cloud_provider": "CLOUD_PROVIDER_AZURE",
  "cluster_type": "TYPE_BYOC",
  "name": "$rp_cluster_name",
  "resource_group_id": "${REDPANDA_RG_ID}",
  "region": "${AZURE_REGION}",
   "customer_managed_resources": {
      "azure" : {
         "management_bucket" : {
            "storage_account_name": "${REDPANDA_MANAGEMENT_STORAGE_ACCOUNT}",
            "storage_container_name": "${REDPANDA_MANAGEMENT_STORAGE_CONTAINER}",
            "resource_group" : { "name" : "${AZURE_RESOURCE_GROUP_NAME}" }
         },
         "vnet" : {
            "name" : "${AZURE_VNET_NAME}",
            "resource_group" : { "name" : "${AZURE_RESOURCE_GROUP_NAME}" }
         },
         "subnets" : {
            "rp_0_pods" : { "name" : "$rp_0_pods_subnet_name" },
            "rp_0_vnet" : { "name" : "$rp_0_vnet_subnet_name" },
            "rp_1_pods" : { "name" : "$rp_1_pods_subnet_name" },
            "rp_1_vnet" : { "name" : "$rp_2_vnet_subnet_name" },
            "rp_2_pods" : { "name" : "$rp_2_pods_subnet_name" },
            "rp_2_vnet" : { "name" : "$rp_2_vnet_subnet_name" },
            "rp_connect_pods" : { "name" : "$rp_connect_pods_subnet_name" },
            "rp_connect_vnet" : { "name" : "$rp_connect_vnet_subnet_name" },
            "kafka_connect_pods" : { "name" : "$kafka_connect_pods_subnet_name" },
            "kafka_connect_vnet" : { "name" : "$kafka_connect_vnet_subnet_name" },
            "sys_pods" : { "name" : "$system_pods_subnet_name" },
            "sys_vnet" : { "name" : "$system_vnet_subnet_name" },
            "rp_agent" :  { "name" : "$rp_agent_subnet_name" },
            "rp_egress_vnet" : { "name" : "$rp_egress_subnet_name" }
         }
      }
   }
}
EOF

Use the Cloud API to create the network and retrieve the network ID:

export REDPANDA_NETWORK_ID=$(curl -X POST "https://api.redpanda.com/v1beta2/networks" \
 -H "accept: application/json" \
 -H "content-type: application/json" \
 -H "authorization: Bearer ${BEARER_TOKEN}" \
 --data-binary @redpanda-network.json | jq -r '.operation.id')

Create cluster

To create the Redpanda cluster:

Define a JSON file called redpanda-cluster.json that includes cluster information:

Show script

cat > redpanda-cluster.json <<EOF
{
  "cloud_provider": "CLOUD_PROVIDER_AZURE",
  "connection_type": "CONNECTION_TYPE_PRIVATE",
  "name": "$cluster_name",
  "resource_group_id": "$namespace_id",
  "network_id": "$network_id",
  "region": "$region",
  "zones": [ $zones ],
  "throughput_tier": "$tier",
  "type": "TYPE_BYOC",
  "customer_managed_resources" : {
     "azure" : {
       "cidrs" : {
          "aks_service_cidr" : "$aks_subnet_cidr"
       },
       "key_vaults": {
          "console_vault": { "name": "$redpanda_console_key_vault_name" },
          "management_vault": { "name": "$redpanda_management_key_vault_name" }
       },
       "resource_groups" : {
          "iam_resource_group" : { "name": "$redpanda_iam_resource_group_name" },
          "redpanda_resource_group" : { "name": "$redpanda_resource_group_name" },
          "storage_resource_group" : { "name": "$redpanda_storage_resource_group_name" }
       },
       "security_groups" : {
          "redpanda_security_group" : { "name": "$redpanda_security_group_name" }
       },
       "tiered_cloud_storage" : {
         "storage_account_name": "$redpanda_tiered_storage_account_name",
         "storage_container_name": "$redpanda_tiered_storage_container_name"
       },
       "user_assigned_identities" : {
         "agent_user_assigned_identity": { "name" : "$agent_user_assigned_identity_name" },
         "aks_user_assigned_identity": { "name" : "$aks_user_assigned_identity_name" },
         "cert_manager_assigned_identity": { "name" : "$cert_manager_assigned_identity_name" },
         "external_dns_assigned_identity": { "name" : "$external_dns_assigned_identity_name" },
         "redpanda_cluster_assigned_identity": { "name" : "$redpanda_cluster_assigned_identity_name" },
         "redpanda_console_assigned_identity": { "name" : "$redpanda_console_assigned_identity_name" },
         "kafka_connect_assigned_identity": { "name" : "$kafka_connect_assigned_identity_name" }
       }
     }
  }
}
EOF

See the full list of zones and tiers available with each provider in the API reference.

Make a Cloud API call to create a Redpanda network and get the network ID from the response in JSON .operation.metadata.network_id.

export REDPANDA_ID=$(curl -X POST "https://api.redpanda.com/v1beta2/clusters" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer ${BEARER_TOKEN}" \
 --data-binary @redpanda-cluster.json | jq -r '.operation.resource_id')

Create cluster resources

To create the initial cluster resources, first log in to Redpanda Cloud with rpk cloud login, and then run rpk cloud byoc azure apply:

rpk cloud login \
  --save \
  --client-id=${REDPANDA_CLIENT_ID} \
  --client-secret=${REDPANDA_CLIENT_SECRET} \
  --no-profile

rpk cloud byoc azure apply --redpanda-id="${REDPANDA_ID}" --subscription-id="${AZURE_SUBSCRIPTION_ID}"

The Redpanda Cloud agent now is running and handles the remaining steps. This can take up to 45 minutes. When provisioning completes, the cluster status updates to Running. If the cluster remains in Creating status after 45 minutes, contact Redpanda Support.

Check cluster status

Cluster creation is an example of an operation that can take a longer period of time to complete. You can check the operation state with the Cloud API, or check the Redpanda Cloud UI for cluster status.

Example using the returned operation_id:

curl -X GET "https://api.redpanda.com/v1beta2/operations/<operation_id of operation from previous step>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer ${BEARER_TOKEN}"

Example retrieving cluster:

curl -X GET "https://api.redpanda.com/v1beta2/clusters/<resource_id of cluster from previous step>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer ${BEARER_TOKEN}"

Delete cluster

To delete the cluster, first send a DELETE request to the Cloud API, and retrieve the resource_id of the DELETE operation. Then run the rpk command to destroy the cluster identified by the resource_id.

export REDPANDA_ID=$(curl -X DELETE "https://api.redpanda.com/v1beta2/clusters/${REDPANDA_ID}" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer ${BEARER_TOKEN}" | jq -r '.operation.resource_id')

After that completes, run:

rpk cloud byoc azure destroy --redpanda-id ${REDPANDA_ID}

Was this helpful?

group Ask in the community

mail Share your feedback

group_add Make a contribution

What do you like about this doc?

Let us know what we do well:

Let us contact you about your feedback:

What did you not like about this doc?

Let us know what we can improve:

Let us contact you about your feedback:

Create a BYOVPC Cluster on Azure

Prerequisites

Limitations

Set environment variables

Create Azure resource group and VNet

Configure Terraform

Deploy Terraform

Authenticate with Redpanda Cloud

Create network

Create cluster

Create cluster resources

Check cluster status

Delete cluster

Simple online edits

Contribution guide