Configure AWS PrivateLink in the Cloud UI

To unlock this feature for your account, contact Redpanda Support.
This guide is for configuring AWS PrivateLink using the Redpanda Cloud UI. To configure and manage PrivateLink on an existing public cluster, you must use the Redpanda Cloud API.

The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Consider using the endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management:

  • PrivateLink allows overlapping CIDR ranges in VPC networks.

  • PrivateLink does not limit the number of connections that use the endpoint service.

  • You control which AWS principals are allowed to connect to the endpoint service.

Requirements

  • Your Redpanda cluster and VPC must be in the same region.

  • Use the AWS CLI to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

Enable endpoint service for existing clusters

  1. In the Redpanda Cloud UI, open your cluster, and click Cluster settings.

  2. Under AWS PrivateLink, click Enable.

  3. You need the Amazon Resource Names (ARNs) for the AWS principals allowed to access the endpoint service. For example, for all principals in a specific account, use arn:aws:iam::<account-id>:root. See the AWS documentation on configuring an endpoint service for details.

  4. It may take several minutes for your cluster to update. When the update is complete, the AWS PrivateLink status in Cluster settings changes from In progress to Enabled.

For help with issues when enabling PrivateLink, contact Redpanda support.

Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud UI.

You can access Redpanda services such as Schema Registry and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default bootstrap port

Kafka API

30292

HTTP Proxy

30282

Schema Registry

30081

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export REDPANDA_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Test the connection

You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    • rpk

    • curl

    echo 'hello world' | rpk topic produce test-topic
    curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

  4. Consume from the test topic.

    • rpk

    • curl

    rpk topic consume test-topic -n 1
    curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"

Disable endpoint service

In Cluster settings, click Disable. Existing connections are closed after the AWS PrivateLink service is disabled. To connect using PrivateLink again, you must re-enable the service.

Suggested reading