Configure Azure Private Link in the Cloud UI

This guide is for configuring new clusters with Azure Private Link using the Redpanda Cloud UI. To configure and manage Private Link on an existing cluster, you must use the Cloud API.

The Redpanda Azure Private Link service provides secure access to Redpanda Cloud from your own VNet. Traffic over Private Link does not go through the public internet because these connections are treated as their own private Azure service. While your VNet has access to the Redpanda virtual network, Redpanda cannot access your VNet.

Consider using the endpoint service if you have multiple VNets and could benefit from a more simplified approach to network management:

  • Azure Private Link allows overlapping CIDR ranges.

  • You control which Azure subscriptions are allowed to connect to the endpoint service.

Requirements

  • Your Redpanda cluster and VNet must be in the same region.

  • Use the Azure command-line interface (CLI) to create a new client VNet or modify an existing one to use the Private Link endpoint.

In Kafka clients, set connections.max.idle.ms to a value less than 350 seconds.

Enable endpoint service for new clusters

  1. In the Redpanda Cloud UI, create a new cluster.

  2. On the Networking page:

    1. For Connection type, select Private.

    2. For Azure Private Link, select Enabled.

    3. For Allowed subscriptions, click Add subscription, and enter the Azure subscription ID that can access the cluster. You can add multiple subscriptions.

Access Redpanda services through VNet endpoint

To access Redpanda services, follow the steps on the cluster’s Overview page. In the How to connect section, click Private Link.

Private Link tab in Overview page

You can access Redpanda services such as Schema Registry and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default bootstrap port

Kafka API

30292

HTTP Proxy

30282

Schema Registry

30081

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Test the connection

You can test the connection to the endpoint service from any VM or container in the consumer VNet. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    • rpk

    • curl

    echo 'hello world' | rpk topic produce test-topic
    curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

  4. Consume from the test topic.

    • rpk

    • curl

    rpk topic consume test-topic -n 1
    curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"