Cloud

Configure Azure Private Link with the Cloud API

For UI-based configuration of Azure Private Link on new clusters, see Configure Azure Private Link in the Cloud Console.

The Redpanda Azure Private Link service provides secure access to Redpanda Cloud from your own virtual network. Traffic over Azure Private Link does not go through the public internet, but instead through Microsoft’s backbone network. While clients can initiate connections against the Redpanda Cloud cluster endpoints, Redpanda Cloud services cannot access your virtual networks directly.

Consider using Private Link if you have multiple virtual networks and require more secure network management. To learn more, see the Azure documentation.

  • Each client VNet can have one endpoint connected to the Private Link service.

  • Private Link allows overlapping CIDR ranges in virtual networks.

  • The number of connections is limited only by your Redpanda usage tier. Private Link does not add extra connection limits.

After getting an access token, you can enable Private Link when creating a new cluster, or you can enable Private Link for existing clusters.

Requirements

  • Install rpk.

  • Install jq, which is used to parse JSON values from API responses.

  • You will use the Azure CLI to authenticate with Azure and configure resources in your Azure account.

  • You will use the Redpanda Cloud API to enable the Redpanda Private Link service for your clusters. Follow the steps on this page to get an access token.

In Kafka clients, set connections.max.idle.ms to a value less than 240 seconds.

Get a Cloud API access token

  1. Save the base URL of the Redpanda Cloud API in an environment variable:

    export PUBLIC_API_ENDPOINT="https://api.cloud.redpanda.com"

  2. In the Redpanda Cloud UI, go to the Organization IAM page, and select the Service account tab. If you don’t have an existing service account, you can create a new one.

    Copy and store the client ID and secret.

    export CLOUD_CLIENT_ID=<client-id>
export CLOUD_CLIENT_SECRET=<client-secret>

  3. Get an API token using the client ID and secret. You can click the Request an API token link to see code examples to generate the token.

    export AUTH_TOKEN=`curl -s --request POST \
    --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data grant_type=client_credentials \
    --data client_id="$CLOUD_CLIENT_ID" \
    --data client_secret="$CLOUD_CLIENT_SECRET" \
    --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`

You must send the API token in the Authorization header when making requests to the Cloud API.

Specify Azure subscriptions

Set the Azure subscriptions you want to use for the Private Link connection.

Replace these placeholder variables:

  • <redpanda-cluster-subscription-id>: The ID of the subscription where the Redpanda cluster is provisioned.

  • <source-connection-subscription-id>: The ID of the subscription from where you initiate connections to the Private Link service.

You may use the same subscription for both.

export REDPANDA_CLUSTER_SUBSCRIPTION_ID=<redpanda-cluster-subscription-id>
export SOURCE_CONNECTION_SUBSCRIPTION_ID=<source-connection-subscription-id>

If you have not yet created a cluster in Redpanda Cloud, create a Private Link-enabled cluster. If you already have a cluster where you want to use Private Link, see the steps to enable Private Link for existing clusters.

  1. In the Redpanda Cloud Console, go to Resource groups and select the Redpanda Cloud resource group in which you want to create a cluster.

    Redpanda Cloud resource groups exist in your Redpanda Cloud account only. They do not correspond to Azure resource groups and do not appear in your Azure tenant.

    Copy and store the resource group ID (UUID) from the URL in the browser.

    export RESOURCE_GROUP_ID=<uuid>

  2. Call POST /v1/networks to create a Redpanda Cloud network for the cluster.

    Make sure to supply your own values in the following example request. Store the network ID (network_id) after the network is created to check whether you can proceed to cluster creation.

    • cluster-type: TYPE_BYOC or TYPE_DEDICATED

    • network-name

    • cidr_block

    • azure-region

    REGION=<azure-region>

NETWORK_POST_BODY=`cat << EOF
{
    "network": {
        "cloud_provider": "CLOUD_PROVIDER_AZURE",
        "cluster_type": "<cluster-type>",
        "name": "<network-name>",
        "cidr_block": "<10.0.0.0/20>",
        "resource_group_id": "$RESOURCE_GROUP_ID",
        "region": "$REGION"
    }
}
EOF`

NETWORK_ID=`curl -vv -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$NETWORK_POST_BODY" $PUBLIC_API_ENDPOINT/v1/networks | jq .metadata.network_id`

echo $NETWORK_ID

    Wait for the network to be ready before creating the cluster in the next step. Check the state of the network creation by calling GET /v1/networks/{id}. You can create the cluster when the state is STATE_READY.

  3. Create a new cluster with the Private Link service enabled by calling POST /v1/clusters.

    In the following example, make sure to set your own values for the following fields:

    CLUSTER_POST_BODY=`cat << EOF
{
  "cluster": {
    "cloud_provider": "CLOUD_PROVIDER_AZURE",
    "connection_type": "CONNECTION_TYPE_PRIVATE",
    "name": "<name>",
    "resource_group_id": "$RESOURCE_GROUP_ID",
    "network_id": "$NETWORK_ID",
    "region": "$REGION",
    "throughput_tier": "<tier>",
    "type": "<type>",
    "zones": [ <zones> ],
    "azure_private_link": {
        "allowed_subscriptions": ["$SOURCE_CONNECTION_SUBSCRIPTION_ID"],
        "enabled": true,
        "connect_console": true
    }
  }
}
EOF`

CLUSTER_ID=`curl -vv -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$CLUSTER_POST_BODY" $PUBLIC_API_ENDPOINT/v1/clusters | jq -r .operation.metadata.cluster_id`

echo $CLUSTER_ID

  4. BYOC clusters only: Check that the cluster operation is completed by calling GET /v1/operations/{id}, and passing the operation ID returned from the Create Cluster call.

    When the Create Cluster operation is completed (STATE_COMPLETED), run the following rpk cloud command to finish setting up your BYOC cluster with Private Link enabled:

    rpk cloud byoc azure apply --redpanda-id=$CLUSTER_ID --subscription-id=$REDPANDA_CLUSTER_SUBSCRIPTION_ID

  5. Continue to configure the Private Link connection to Redpanda.

Enabling Private Link on your VNet interrupts all communication on existing Redpanda bootstrap server and broker ports due to the change of private DNS resolution. Make sure all applications running in your virtual network are ready to start using the corresponding Private Link ports.

  1. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the Details section.

    CLUSTER_ID=<cluster_id>

  2. Make a PATCH /v1/clusters/{cluster.id} request to update the cluster with the service enabled.

    CLUSTER_PATCH_BODY=`cat << EOF
{
  "azure_private_link": {
      "allowed_subscriptions": ["$SOURCE_CONNECTION_SUBSCRIPTION_ID"],
      "enabled": true,
      "connect_console": true
  }
}
EOF`

curl -vv -X PATCH \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID

  3. Before proceeding, check the state of the Update Cluster operation by calling GET /v1/operations/{id}, and passing the operation ID returned from the Update Cluster call. When the state is STATE_READY, continue to configure the Private Link connection to Redpanda.

  1. In the Redpanda Cloud Console, go to Users and create a new user to authenticate the Private Link endpoint connections with the service. You will need the username and password to access Redpanda services or test the connection using rpk or cURL.

  2. Call the GET /v1/clusters/{id} endpoint to check the service status and retrieve the service ID, DNS name, and Redpanda Console URL to use.

    DNS_RECORD=`curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | jq -r ".cluster.azure_private_link.status.dns_a_record"`

PRIVATE_SERVICE_ID=`curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | jq -r ".cluster.azure_private_link.status.service_id"`

CONSOLE_URL=`curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | jq -r ".cluster.redpanda_console.url"`

echo $DNS_RECORD
echo $PRIVATE_SERVICE_ID
echo $CONSOLE_URL

  3. Log in to Azure and set the subscription ID to the value you set for SOURCE_CONNECTION_SUBSCRIPTION_ID:

    az login
az account set --subscription $SOURCE_CONNECTION_SUBSCRIPTION_ID

  1. If you have not already done so, create the Azure resource group and virtual network for your Private Link source connections.

    az group create --name <azure-pl-endpoint-resource-group-name> --location $REGION
    az network vnet create \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --location $REGION \
    --name <azure-pl-endpoint-vnet-name> \
    --address-prefixes 10.0.0.0/16 \
    --subnet-name <subnet-name> \
    --subnet-prefixes 10.0.0.0/24

  2. Create the private endpoint.

    az network private-endpoint create \
    --location $REGION \
    --connection-name <private-link-service-name> \
    --name redpanda-$CLUSTER_ID \
    --manual-request true \
    --private-connection-resource-id $PRIVATE_SERVICE_ID \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --subnet <subnet-name> \
    --vnet-name <azure-pl-endpoint-vnet-name>

  3. Create a private DNS zone using the outputted DNS record above (echo $DNS_RECORD)

    az network private-dns zone create \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --name "$DNS_RECORD"

  4. Link the private DNS zone to the virtual network you created earlier, so virtual machines (VMs) and containers can resolve the Redpanda cluster domain.

    az network private-dns link vnet create \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --zone-name "$CLUSTER_ID.byoc.prd.cloud.redpanda.com" \
    --name redpanda-$CLUSTER_ID-dns-zone-link \
    --virtual-network <azure-pl-endpoint-vnet-name> \
    --registration-enabled false

  5. Create a wildcard record in the private DNS zone.

    az network private-dns record-set a add-record \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --zone-name redpanda-$CLUSTER_ID \
    --record-set-name "*" \
    --ipv4-address "$PRIVATE_ENDPOINT_IP"

After you enable Private Link for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

You can access Redpanda services such as Redpanda Console, Schema Registry, and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default port

Kafka API

30292

HTTP Proxy

30282

Schema Registry

30081

Redpanda Console

443

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Verify the Redpanda Console network path

When you configure private connectivity through the Cloud API, set connect_console: true on the network configuration to enable Console access through the cluster’s private endpoint. The Cloud UI sets this for you when you enable private connectivity.

The Redpanda Console URL is served on port 443 (HTTPS) and follows the form https://console-<id>.<cluster_domain>;. The <id> is a per-cluster suffix assigned by the Redpanda control plane and is opaque to you; the full URL is shown in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

To verify that the network path to Redpanda Console is open, run the following commands from a host in the client network. First, confirm DNS resolves the hostname to a private IP on the cluster’s private endpoint:

dig +short console-<id>.<cluster_domain>

The response is a private IP from your client network’s address range, for example:

10.0.0.42

Then confirm Redpanda Console responds over HTTPS through the endpoint:

curl -sS -o /dev/null -w "%{http_code}\n" https://console-<id>.<cluster_domain>/

Expected output:

200

A 200 response confirms the network path. The Console UI itself does not expose a standalone login form: sign in to cloud.redpanda.com, navigate to the cluster, and use the cluster’s left navigation (Topics, Brokers, Consumer groups) to interact with Redpanda Console.

  • DNS resolution for the Redpanda Console hostname is handled automatically by the cluster’s private endpoint. You don’t need to create a private hosted zone or override DNS in the client network.

  • Ensure your network access rules (for example, AWS security groups, Azure NSGs, or GCP firewall rules) on the private endpoint allow inbound TCP on port 443 from your client workload sources only (for example, the client network’s CIDR or specific client access groups). Avoid broad source ranges such as 0.0.0.0/0.

Test the connection

You can test the Private Link connection from any VM or container in the subscription where the endpoint is created. If configuring a Kafka client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    • rpk

    • curl

    echo 'hello world' | rpk topic produce test-topic
    curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

  4. Consume from the test topic.

    • rpk

    • curl

    rpk topic consume test-topic -n 1
    curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"