Create a BYOC Cluster on Azure

To create a Redpanda cluster in your virtual network (VNet), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run rpk cloud byoc apply. See also: BYOC architecture.

With standard BYOC clusters, Redpanda manages security policies and resources for your virtual network (VNet), including subnetworks, managed identities, IAM roles, security groups, and storage accounts. For the most security, you can manage these resources yourself with a BYOVNet cluster on Azure.

Prerequisites

Before you deploy a BYOC cluster on Azure, check all prerequisites to ensure that your Azure subscription meets requirements.

Configure Azure CLI

  • Install the Azure CLI.

  • Sign in with the Azure CLI:

    az login

  • Set the desired subscription for the Azure CLI:

    az account set --subscription <subscription-name>

Verify rpk version

Confirm you have a minimum version of Redpanda rpk v24.1. See Install or Update rpk.

Prepare your Azure subscription

In the Azure Portal, confirm that the dedicated subscription you intend to use with Redpanda includes the following:

  • Role: The Azure user must have the Owner role in the subscription.

  • Resources: The subscription must be registered for the following resource providers. See the Microsoft documentation.

    • Microsoft.Compute

    • Microsoft.ManagedIdentity

    • Microsoft.Storage

    • Microsoft.KeyVault

    • Microsoft.Network

    To check if a resource provider is registered, run the following command using the Azure CLI or in the Azure Cloud Shell. For example, to check for Microsoft.Compute, run:

    az provider show -n Microsoft.Compute

    If it is not registered, run:

    az provider register --namespace 'Microsoft.Compute'

  • Feature: The subscription must be registered for Microsoft.Compute/EncryptionAtHost. See the Microsoft documentation.

    To register it, run:

    az feature register --namespace Microsoft.Compute --name EncryptionAtHost

  • Monitoring: The subscription must have Azure Network Watcher enabled in the NetworkWatcherRG resource group and the region where you will use Redpanda. Network Watcher lets you monitor and diagnose conditions at a network level. See the Microsoft documentation.

    To enable it, run:

    # Create the NetworkWatcherRG resource group
az group create --name 'NetworkWatcherRG' --location '<region_name>'

# Enable Network Watcher in <region_name>
az network watcher configure --resource-group 'NetworkWatcherRG' --locations '<region_name>' --enabled

Check Azure quota

Confirm that the Azure subscription has enough virtual CPUs (vCPUs) per instance family and total regional vCPUs in the region where you will use Redpanda:

  • Standard Ddv5-series vCPUs: 12 (3 Redpanda broker nodes + extra capacity for 3 more nodes that could be utilized temporarily during tier 1 maintenance)

  • Standard Dadsv5-series vCPUs: 8 (2 Redpanda utility nodes)

  • Standard Dv3-series vCPUs: 2 (1 Redpanda agent node)

See the Microsoft documentation.

Check Azure SKU restrictions

Ensure your subscription has access to the required VM sizes in the region where you will use Redpanda. For example, using the Azure CLI or in the Azure Cloud Shell, run:

# Replace eastus2 with your target region
az vm list-skus -l eastus2 --zone --size Standard_D2d_v5 --output table
Example output (no restrictions: good)
ResourceType     Locations    Name               Zones    Restrictions
---------------  -----------  ---------------    -------  ------------
virtualMachines  eastus2      Standard_D2d_v5    1,2,3    None
Example output (with restrictions: needs attention)
ResourceType     Locations    Name               Zones    Restrictions
---------------  -----------  ---------------    -------  ------------
virtualMachines  eastus2      Standard_D2d_v5    1,2,3    NotAvailableForSubscription

If you see restrictions, open a Microsoft support request to remove them.

Prerequisite checklist

  • Verified rpk version

  • Verified Azure user has Owner role

  • Registered all required resource providers

  • Registered EncryptionAtHost feature

  • Enabled Network Watcher

  • Verified vCPU quota

  • Verified no SKU restrictions

Create a BYOC cluster

To create a Redpanda cluster in your Azure VNet, follow the prerequisites then follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run rpk cloud byoc apply.

  1. Log in to Redpanda Cloud.

  2. On the Clusters page, click Create cluster, then click Create BYOC cluster.

  3. Enter a cluster name, then select the resource group, provider (Azure), region, tier, availability, and Redpanda version.

    • If you plan to create a private network in your own VNet, select the region where your VNet is located.

    • Multi-AZ is the default configuration. Three AZs provide two backups in case one availability zone goes down.

    Optionally, click Advanced settings to specify up to five key-value custom tags. After the cluster is created, the tags are applied to all Azure resources associated with this cluster. For details, see the Microsoft documentation. After the cluster is created, you can specify more tags with the Cloud API.

  4. Click Next.

  5. On the Network page, enter the connection type: either Public or Private. For BYOC clusters, Private using Azure Private Link is best-practice.

    • Your network name is used to identify this network.

    • For a CIDR range, choose one that does not overlap with your existing VPCs or your Redpanda network.

  6. Click Next.

  7. On the Deploy page, follow the steps to log in to Redpanda Cloud and deploy the agent.

    As part of agent deployment, Redpanda assigns the permissions required to run the agent. For details about these permissions, see Azure IAM policies.

Manage custom tags

Your organization might require custom tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage tags with the Cloud Control Plane API. The Control Plane API allows up to 16 custom tags in Azure.

Make sure you have:

  • The cluster ID. You can find this in the Redpanda Cloud UI, in the Details section of the cluster overview.

  • A valid bearer token for the Cloud Control Plane API. For details, see Authenticate to the API.

To unlock this feature for your account, contact Redpanda Support.

  1. To refresh Redpanda agent permissions in the target subscription, run:

    export CLUSTER_ID="<cluster-id>"
export SUBSCRIPTION_ID="<subscription-id>"

rpk cloud byoc azure apply --redpanda-id="$CLUSTER_ID" --subscription-id="$SUBSCRIPTION_ID"

  2. To update tags, invoke the Cloud API.

    First, set your authentication token:

    export AUTH_TOKEN="<your-bearer-token>"

    The PATCH call sets the tags specified under "cloud_provider_tags". It replaces the existing tags with the specified tags. Include all desired tags in the request. To remove a single entry, omit it from the map you send.

    cluster_patch_body=$(cat <<'JSON'
{
  "cloud_provider_tags": {
    "Environment": "production",
    "CostCenter": "engineering"
  }
}
JSON
)

curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d "$cluster_patch_body"

    To remove all tags, send an empty cloud_provider_tags object:

    cluster_patch_body='{"cloud_provider_tags": {}}'

curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d "$cluster_patch_body"

Limitations

  • Nodepool Application Security Groups (ASG): Custom tags are set only when the cluster is created. Tags cannot be updated on these resources after cluster creation.

  • Private Link network interfaces (Kubernetes API server, Tiered Storage, and Private Link service): Custom tags are set only during cluster creation and cannot be changed later.