Create a Customer-Managed VNet on Azure

To unlock this feature for your account, contact Redpanda support.

With a standard BYOC cluster, Redpanda manages the networking lifecycle. For additional security, you can deploy the Redpanda data plane into your existing virtual network (VNet) and manage the lifecycle yourself. When you create a customer-managed VNet cluster, you specify your VNet and managed identities. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With a customer-managed VNet:

You provide your own VNet in your Azure account.
You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters.
You control your security resources and policies, including subnets, user-assigned identities, IAM roles and assignments, security groups, storage accounts, and key vaults.

The Redpanda repository contains the code that deploys the resources you must create for a customer-managed Azure cluster. You create these resources in advance and provide them to Redpanda during cluster creation. The code is provided in Terraform. There may be resources in the repository that already exist in your environment that you don’t want to create (for example, the VNet). Variables are provided for this purpose.

See the code for the complete list of resources required to create and deploy Redpanda cluster. Customer-managed resources can be broken down into the following groups:

Resource group resources
User-assigned identities
IAM roles and assignments
Network
Storage
Key vaults

Prerequisites

Access to an Azure subscription where you want to create your cluster
Knowledge of your internal VNet and subnet configuration
Permission to call the Redpanda Cloud API
Permission to create, modify, and delete the resources described by Terraform
Terraform version 1.8.5 or later

Limitations

You cannot move existing clusters to a customer-managed VNet.
You cannot change to a different VNet after creating a cluster with a customer-managed VNet.

Deploy customer-managed resources With Terraform

You may want to configure remote state for this Terraform. For simplicity, these instructions assume local state.

Configure the variables

Create a JSON file called byovnet.auto.tfvars.json inside the Terraform directory to configure variables for your specific needs:

{
  "azure_tenant_id": "",
  "azure_subscription_id": "",
  "region": "",
  "resource_name_prefix": "",
  "tags": {},
  "redpanda_resource_group_name": "",
  "redpanda_storage_resource_group_name": "",
  "redpanda_network_resource_group_name": "",
  "redpanda_iam_resource_group_name": "",
  "redpanda_agent_identity_name": "",
  "redpanda_external_dns_identity_name": "",
  "redpanda_cluster_identity_name": "",
  "redpanda_cert_manager_identity_name": "",
  "aks_identity_name": "",
  "redpanda_console_identity_name": "",
  "kafka_connect_identity_name": "",
  "redpanda_management_storage_account_name": "",
  "redpanda_management_storage_container_name": "",
  "redpanda_tiered_storage_account_name": "",
  "redpanda_tiered_storage_container_name": "",
  "redpanda_management_key_vault_name": "",
  "redpanda_console_key_vault_name": "",
  "vnet_name": "",
  "vnet_addresses": "10.0.0.0/20",
  "private_subnets": {},
  "egress_subnets": {},
  "reserved_subnet_cidrs": {},
  "redpanda_security_group_name": ""
}

The Terraform Readme has more information about the variable files:

vars.azure.tf: Contains the variables related to Azure credentials.
vars.condition.tf: Contains the conditional variables; for example, on whether to create NAT.
vars.customer_input.tf: Contains the variables needed to create a cluster with the Cloud API.
vars.iam.tf: Contains the variables related to IAM roles.
vars.misc.tf: Contains the variables, regions, zones, tags, and resource name prefix.

Initialize Terraform

Initialize the working directory containing Terraform configuration files:

terraform init

Apply Terraform

terraform apply

The output of terraform apply displays output values. Note these values, because they are needed in subsequent steps. To continue with cluster creation, see Create Azure VNet Redpanda Cluster.

You can also get these values by running terraform output. For example:

agent_private_subnet_name = "my-snet-agent-private"
agent_user_assigned_identity_name = "my-agent-uai"
aks_user_assigned_identity_name = "my-aks-uai"
cert_manager_user_assigned_identity_name = "my-cert-manager-uai"
cluster_user_assigned_identity_name = "my-cluster-uai"
console_key_vault_name = "my-consolevault"
console_user_assigned_identity_name = "my-console-uai"
egress_subnet_name = "my-snet-agent-public"
external_dns_user_assigned_identity_name = "my-external-dns-uai"
iam_resource_group_name = "my-iam-rg"
kafka_connect_pods_subnet_name = "my-snet-kafka-connect-pods"
kafka_connect_user_assigned_identity_name = "my-kafka-connect-uai"
kafka_connect_vnet_subnet_name = "my-snet-kafka-connect-vnet"
management_bucket_storage_account_name = "mymanagement"
management_bucket_storage_container_name = "mymanagement"
management_key_vault_name = "my-redpandavault"
network_resource_group_name = "my-network-rg"
......
redpanda_resource_group_name = "my-redpanda-rg"
redpanda_security_group_name = "my-redpanda-nsg"
......
rp_0_pods_subnet_name = "my-snet-rp-0-pods"
rp_0_vnet_subnet_name = "my-snet-rp-0-vnet"
rp_1_pods_subnet_name = "my-snet-rp-1-pods"
rp_1_vnet_subnet_name = "my-snet-rp-1-vnet"
rp_2_pods_subnet_name = "my-snet-rp-2-pods"
rp_2_vnet_subnet_name = "my-snet-rp-2-vnet"
rp_connect_pods_subnet_name = "my-snet-connect-pods"
rp_connect_vnet_subnet_name = "my-snet-connect-vnet"
......
storage_resource_group_name = "my-storage-rg"
system_pods_subnet_name = "my-snet-system-pods"
system_vnet_subnet_name = "my-snet-system-vnet"
tiered_storage_account_name = "mytieredstorage"
tiered_storage_container_name = "mytieredstorage"
......
vnet_name = "my-rp-vnet"

Create a network with the Cloud API

Create a network with the POST body, and replace the variables with your actual values. To create a resource group and bearer token, follow the Cloud API Quickstart.

network_post_body=`cat << EOF
{
  "cloud_provider": "CLOUD_PROVIDER_AZURE",
  "cluster_type": "TYPE_BYOC",
  "name": "$rp_cluster_name",
  "resource_group_id": "$resource_group_id",
  "region": "$region",
   "customer_managed_resources": {
      "azure" : {
         "management_bucket" : {
            "storage_account_name": "$redpanda_management_storage_account_name",
            "storage_container_name": "$redpanda_management_storage_container_name",
            "resource_group" : { "name" : "$redpanda_resource_group_name" }
         },
         "vnet" : {
            "name" : "$vnet_name",
            "resource_group" : { "name" : "$redpanda_network_resource_group_name" }
         },
         "subnets" : {
            "rp_0_pods" : { "name" : "$rp_0_pods_subnet_name" },
            "rp_0_vnet" : { "name" : "$rp_0_vnet_subnet_name" },
            "rp_1_pods" : { "name" : "$rp_1_pods_subnet_name" },
            "rp_1_vnet" : { "name" : "$rp_2_vnet_subnet_name" },
            "rp_2_pods" : { "name" : "$rp_2_pods_subnet_name" },
            "rp_2_vnet" : { "name" : "$rp_2_vnet_subnet_name" },
            "rp_connect_pods" : { "name" : "$rp_connect_pods_subnet_name" },
            "rp_connect_vnet" : { "name" : "$rp_connect_vnet_subnet_name" },
            "kafka_connect_pods" : { "name" : "$kafka_connect_pods_subnet_name" },
            "kafka_connect_vnet" : { "name" : "$kafka_connect_vnet_subnet_name" },
            "sys_pods" : { "name" : "$system_pods_subnet_name" },
            "sys_vnet" : { "name" : "$system_vnet_subnet_name" },
            "rp_agent" :  { "name" : "$rp_agent_subnet_name" },
            "rp_egress_vnet" : { "name" : "$rp_egress_subnet_name" }
         }
      }
   }
}
EOF`

Make a Cloud API call to create a Redpanda network and get the network ID from the response in JSON .operation.metadata.network_id.

curl -vv -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $token" \
  -d "$network_post_body" $PUBLIC_API_ENDPOINT/v1beta2/networks

Create a cluster with the Cloud API

A network ID is required to create a Redpanda cluster. Replace the variables with your actual values. To create a resource group and bearer token, follow the Cloud API Quickstart.

cluster_post_body=`cat << EOF
{
  "cloud_provider": "CLOUD_PROVIDER_AZURE",
  "connection_type": "CONNECTION_TYPE_PRIVATE",
  "name": "$cluster_name",
  "resource_group_id": "$namespace_id",
  "network_id": "$network_id",
  "region": "$region",
  "zones": [ $zones ],
  "throughput_tier": "$tier",
  "type": "TYPE_BYOC",
  "customer_managed_resources" : {
     "azure" : {
       "cidrs" : {
          "aks_service_cidr" : "$aks_subnet_cidr"
       },
       "key_vaults": {
          "console_vault": { "name": "$redpanda_console_key_vault_name" },
          "management_vault": { "name": "$redpanda_management_key_vault_name" }
       },
       "resource_groups" : {
          "iam_resource_group" : { "name": "$redpanda_iam_resource_group_name" },
          "redpanda_resource_group" : { "name": "$redpanda_resource_group_name" },
          "storage_resource_group" : { "name": "$redpanda_storage_resource_group_name" }
       },
       "security_groups" : {
          "redpanda_security_group" : { "name": "$redpanda_security_group_name" }
       },
       "tiered_cloud_storage" : {
         "storage_account_name": "$redpanda_tiered_storage_account_name",
         "storage_container_name": "$redpanda_tiered_storage_container_name"
       },
       "user_assigned_identities" : {
         "agent_user_assigned_identity": { "name" : "$agent_user_assigned_identity_name" },
         "aks_user_assigned_identity": { "name" : "$aks_user_assigned_identity_name" },
         "cert_manager_assigned_identity": { "name" : "$cert_manager_assigned_identity_name" },
         "external_dns_assigned_identity": { "name" : "$external_dns_assigned_identity_name" },
         "redpanda_cluster_assigned_identity": { "name" : "$redpanda_cluster_assigned_identity_name" },
         "redpanda_console_assigned_identity": { "name" : "$redpanda_console_assigned_identity_name" },
         "kafka_connect_assigned_identity": { "name" : "$kafka_connect_assigned_identity_name" }
       }
     }
  }
}
EOF`

Make a Cloud API call to create a Redpanda network and get the network ID from the response in JSON .operation.metadata.network_id.

curl -vv -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $token" \
  -d "$cluster_post_body" $PUBLIC_API_ENDPOINT/v1beta2/clusters

Deploy the cluster

After getting a Redpanda cluster ID, deploy the cluster with rpk, replacing $rp_id and $subscription_id with your values:

rpk cloud byoc azure apply --redpanda-id='$rp_id' --subscription-id='$subscription_id'

Check cluster status

You can check the cluster status with the Cloud API or the Redpanda Cloud UI.

Example using the returned operation_id:

curl -X GET "https://api.redpanda.com/v1beta2/operations/<operation_id of operation from previous step>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer $YOUR_TOKEN"

Example retrieving cluster:

curl -X GET "https://api.redpanda.com/v1beta2/clusters/<resource_id of cluster from previous step>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer $YOUR_TOKEN"

Was this helpful?

group Ask in the community

mail Share your feedback

group_add Make a contribution

What do you like about this doc?

Let us know what we do well:

Let us contact you about your feedback:

What did you not like about this doc?

Let us know what we can improve: