Configure Azure Private Link with the Cloud API

To unlock this feature for your account, contact Redpanda Support.

The Redpanda Azure Private Link service provides secure access to Redpanda Cloud from your own virtual network. Traffic over Azure Private Link does not go through the public internet, but instead through Microsoft’s backbone network. While clients can initiate connections against the Redpanda Cloud cluster endpoints, Redpanda Cloud services cannot access your virtual networks directly.

Consider using Private Link if you have multiple virtual networks and require more secure network management.

Private Link allows overlapping CIDR ranges in virtual networks.
Private Link does not limit the number of connections.

After getting an access token, you can enable Private Link when creating a new cluster, or you can enable Private Link for existing clusters.

Requirements

Install rpk.
You will use the Azure CLI to authenticate with Azure and configure resources in your Azure account.
You will use the Redpanda Cloud API to enable the Redpanda Private Link service for your clusters. Follow the steps on this page to get an access token.

To learn more about Azure Private Link, see the Azure documentation.

Set up Redpanda Private Link Service

Get a Cloud API access token

Save the base URL of the Redpanda Cloud API in an environment variable:
```
export PUBLIC_API_ENDPOINT="https://api.cloud.redpanda.com"
```
In your organization in the Redpanda Cloud UI, go to Clients. If you don’t have an existing client (also known as service account), you can create a new one by clicking Add client.

Copy and store the client ID and secret.
```
export CLOUD_CLIENT_ID=<client-id>
export CLOUD_CLIENT_SECRET=<client-secret>
```

Get an API token using the client ID and secret. You can click the Request an API token link to see code examples to generate the token.

export AUTH_TOKEN=`curl -s --request POST \
    --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data grant_type=client_credentials \
    --data client_id="$CLOUD_CLIENT_ID" \
    --data client_secret="$CLOUD_CLIENT_SECRET" \
    --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`

You must send the API token in the Authorization header when making requests to the Cloud API.

Specify Azure subscriptions

Set the Azure subscriptions you want to use for the Private Link connection.

Replace these placeholder variables:

<redpanda-cluster-subscription-id>: The ID of the subscription where the Redpanda cluster is provisioned.
<source-connection-subscription-id>: The ID of the subscription from where you initiate connections to the Private Link service.

You may use the same subscription for both.

export REDPANDA_CLUSTER_SUBSCRIPTION_ID=<redpanda-cluster-subscription-id>
export SOURCE_CONNECTION_SUBSCRIPTION_ID=<source-connection-subscription-id>

If you have not yet created a cluster in Redpanda Cloud, create a Private Link-enabled cluster. If you already have a cluster where you want to use Private Link, see the steps to enable Private Link for existing clusters.

Create new cluster with Private Link service enabled

In the Redpanda Cloud UI, go to Resource groups and select the Redpanda Cloud resource group in which you want to create a cluster.

Redpanda Cloud resource groups exist in your Redpanda Cloud account only. They do not correspond to Azure resource groups and do not appear in your Azure tenant.

Copy and store the resource group ID (UUID) from the URL in the browser.
```
export RESOURCE_GROUP_ID=<uuid>
```
Call POST /v1beta2/networks to create a Redpanda Cloud network for the cluster.

Make sure to supply your own values in the following example request. Store the network ID (network_id) after the network is created to check whether you can proceed to cluster creation.
- cluster-type: TYPE_BYOC or TYPE_DEDICATED
- network-name
- cidr_block
- azure-region
```
REGION=<azure-region>

NETWORK_POST_BODY=`cat << EOF
{
    "cloud_provider": "CLOUD_PROVIDER_AZURE",
    "cluster_type": "<cluster-type>",
    "name": "<network-name>",
    "cidr_block": "<10.0.0.0/20>",
    "resource_group_id": "$RESOURCE_GROUP_ID",
    "region": "$REGION"
}
EOF`

NETWORK_ID=`curl -vv -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$NETWORK_POST_BODY" $PUBLIC_API_ENDPOINT/v1beta2/networks | jq .metadata.network_id`

echo $NETWORK_ID
```
Wait for the network to be ready before creating the cluster in the next step. Check the state of the network creation by calling GET /v1beta2/networks/{id}. You can create the cluster when the state is STATE_READY.

Create a new cluster with the Private Link service enabled by calling POST /v1beta2/clusters.

In the following example, make sure to set your own values for the following fields:

name
type: TYPE_BYOC or TYPE_DEDICATED
tier: For example, tier-1-azure. See available Azure tiers in the Cloud API reference. To learn more about tiers, see BYOC Tiers and Regions or Dedicated Tiers and Regions.
zones: For example, "uksouth-az1", "uksouth-az2", "uksouth-az3"

CLUSTER_POST_BODY=`cat << EOF
{
  "cloud_provider": "CLOUD_PROVIDER_AZURE",
  "connection_type": "CONNECTION_TYPE_PRIVATE",
  "name": "<name>",
  "resource_group_id": "$RESOURCE_GROUP_ID",
  "network_id": "$NETWORK_ID",
  "region": "$REGION",
  "throughput_tier": "<tier>",
  "type": "<type>",
  "zones": [ <zones> ],
  "azure_private_link": {
      "allowed_subscriptions": ["$SOURCE_CONNECTION_SUBSCRIPTION_ID"],
      "enabled": true,
      "connect_console": true
  }
}
EOF`

CLUSTER_ID=`curl -vv -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$CLUSTER_POST_BODY" $PUBLIC_API_ENDPOINT/v1beta2/clusters | jq -r .operation.metadata.cluster_id`

echo $CLUSTER_ID

BYOC clusters: Check that the cluster operation is completed by calling GET /v1beta2/operations/{id}, and passing the operation ID returned from the Create Cluster call.

When the Create Cluster operation is completed (STATE_COMPLETED), run the following rpk cloud command to finish setting up your BYOC cluster with Private Link enabled:
```
rpk cloud byoc azure apply --redpanda-id=$CLUSTER_ID --subscription-id=$REDPANDA_CLUSTER_SUBSCRIPTION_ID
```
Continue to configure the Private Link connection to Redpanda.

Enable Private Link service for existing clusters

As soon as Private Link is available on your virtual network, all communication on existing Redpanda bootstrap server and broker ports is interrupted due to the change on the private DNS resolution. Make sure all applications running in your virtual network are ready to start using the corresponding Private Link ports.

In the Redpanda Cloud UI, go to the cluster overview and copy the cluster ID from the Details section.
```
CLUSTER_ID=<cluster_id>
```

Make a PATCH /v1beta2/clusters/{cluster.id} request to update the cluster with the service enabled.

CLUSTER_PATCH_BODY=`cat << EOF
{
  "azure_private_link": {
      "allowed_subscriptions": ["$SOURCE_CONNECTION_SUBSCRIPTION_ID"],
      "enabled": true,
      "connect_console": true
  }
}
EOF`

curl -vv -X PATCH \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1beta2/clusters/$CLUSTER_ID

Before proceeding, check the state of the Update Cluster operation by calling GET /v1beta2/operations/{id}, and passing the operation ID returned from the Update Cluster call. When the state is STATE_READY, continue to configure the Private Link connection to Redpanda.

Configure Azure Private Link connection to Redpanda Cloud

In the Redpanda Cloud UI, go to Users and create a new user to authenticate the Private Link endpoint connections with the service. You will need the username and password to access Redpanda services or test the connection using rpk or cURL.

Call the GET /v1beta2/clusters/{id} endpoint to check the service status and retrieve the service ID, DNS name, and Redpanda Console URL to use.

DNS_RECORD=`curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1beta2/clusters/$CLUSTER_ID | jq -r ".cluster.azure_private_link.status.dns_a_record"`

PRIVATE_SERVICE_ID=`curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1beta2/clusters/$CLUSTER_ID | jq -r ".cluster.azure_private_link.status.service_id"`

CONSOLE_URL=`curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1beta2/clusters/$CLUSTER_ID | jq -r ".cluster.redpanda_console.url"`

echo $DNS_RECORD
echo $PRIVATE_SERVICE_ID
echo $CONSOLE_URL

Log in to Azure and set the subscription ID to the value you set for SOURCE_CONNECTION_SUBSCRIPTION_ID:
```
az login
az account set --subscription $SOURCE_CONNECTION_SUBSCRIPTION_ID
```

Set up Azure Private Link endpoint in your virtual network

If you have not already done so, create the Azure resource group and virtual network for your Private Link source connections.

az group create --name <azure-pl-endpoint-resource-group-name> --location $REGION

az network vnet create \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --location $REGION \
    --name <azure-pl-endpoint-vnet-name> \
    --address-prefixes 10.0.0.0/16 \
    --subnet-name <subnet-name> \
    --subnet-prefixes 10.0.0.0/24

Create the private endpoint.

az network private-endpoint create \
    --location $REGION \
    --connection-name <private-link-service-name> \
    --name redpanda-$CLUSTER_ID \
    --manual-request yes \
    --private-connection-resource-id $PRIVATE_SERVICE_ID \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --subnet <subnet-name> \
    --vnet-name <azure-pl-endpoint-vnet-name>

Create a private DNS zone using the outputted DNS record above (echo $DNS_RECORD)

az network private-dns zone create \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --name "$DNS_RECORD"

Link the private DNS zone to the virtual network you created earlier, so virtual machines (VMs) and containers can resolve the Redpanda cluster domain.

az network private-dns link vnet create \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --zone-name "$CLUSTER_ID.byoc.prd.cloud.redpanda.com" \
    --name redpanda-$CLUSTER_ID-dns-zone-link \
    --virtual-network <azure-pl-endpoint-vnet-name> \
    --registration-enabled false

Create a wildcard record in the private DNS zone.

az network dns record-set a add-record \
    --resource-group <azure-pl-endpoint-resource-group-name> \
    --zone-name redpanda-$CLUSTER_ID \
    --record-set-name "*" \
    --ipv4-address "$PRIVATE_ENDPOINT_IP"

Connect to Redpanda services through Private Link endpoints

After you enable Private Link for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud UI.

You can access Redpanda services such as Schema Registry and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service	Default bootstrap port
Kafka API	30292
HTTP Proxy	30282
Schema Registry	30081

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export REDPANDA_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Test the connection

You can test the Private Link connection from any VM or container in the subscription where the endpoint is created. If configuring a Kafka client isn’t possible right away, you can do these checks using rpk or cURL:

Set the following environment variables.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

Create a test topic.
```
rpk topic create test-topic
```

Produce to the test topic.

rpk
curl

echo 'hello world' | rpk topic produce test-topic

curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

Consume from the test topic.

rpk
curl

rpk topic consume test-topic -n 1

curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"

Was this helpful?

group Ask in the community

mail Share your feedback

group_add Make a contribution

What do you like about this doc?

Let us know what we do well:

Let us contact you about your feedback:

What did you not like about this doc?

Let us know what we can improve: