Configure AWS PrivateLink in the Cloud UI
To unlock this feature for your account, contact Redpanda support. |
This guide is for configuring AWS PrivateLink using the Redpanda Cloud UI. See Configure AWS PrivateLink for Redpanda Cloud if you want to set up the endpoint service using the API. |
The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.
Consider using the endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management:
-
PrivateLink allows overlapping CIDR ranges in VPC networks.
-
PrivateLink does not limit the number of connections that use the endpoint service.
-
You control which AWS principals are allowed to connect to the endpoint service.
Requirements
-
Your Redpanda cluster and VPC must be in the same region.
-
Use the AWS CLI to create a new client VPC or modify an existing one to use the PrivateLink endpoint.
Enable endpoint service for existing clusters
-
In the Redpanda Cloud UI, open your cluster, and click Cluster settings.
-
Under AWS PrivateLink, click Enable.
-
You need the Amazon Resource Names (ARNs) for the AWS principals allowed to access the endpoint service. For example, for all principals in a specific account, use
arn:aws:iam::<account-id>:root
. See the AWS documentation on configuring an endpoint service for details. -
It may take several minutes for your cluster to update. When the update is complete, the AWS PrivateLink status in Cluster settings changes from In progress to Enabled.
For help with issues when enabling PrivateLink, contact Redpanda support. |
Access Redpanda services through VPC endpoint
After you have enabled PrivateLink for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud UI.
You can access Redpanda services such as Schema Registry and HTTP Proxy from the client VPC, for example, from a compute instance in the VPC.
The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:
Redpanda service | Default bootstrap port |
---|---|
Kafka API |
30292 |
HTTP Proxy |
30282 |
Schema Registry |
30081 |
Access Kafka API seed service
Use port 30292
to access the Kafka API seed service.
export REDPANDA_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>
When successful, the rpk
output should look like the following:
CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g
BROKERS
=======
ID HOST PORT RACK
0* 0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com 32092 use2-az1
1 1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com 32093 use2-az1
2 2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com 32094 use2-az1
Test the connection
You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn’t possible right away, you can do these checks using rpk
or cURL:
-
Set the following environment variables.
export REDPANDA_BROKERS='<kafka-api-bootstrap-server-hostname>:30292' export RPK_TLS_ENABLED=true export RPK_USER=<user> export RPK_PASS=<password>
-
Create a test topic.
rpk topic create test-topic
-
Produce to the test topic.
-
rpk
-
curl
echo 'hello world' | rpk topic produce test-topic
curl -s \ -X POST \ "<http-proxy-bootstrap-server-url>/topics/test-topic" \ -H "Content-Type: application/vnd.kafka.json.v2+json" \ -d '{ "records":[ { "value":"hello world" } ] }'
-
-
Consume from the test topic.
-
rpk
-
curl
rpk topic consume test-topic -n 1
curl -s \ "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\ -H "Accept: application/vnd.kafka.json.v2+json"
-
Disable endpoint service
In Cluster settings, click Disable. Existing connections are closed after the AWS PrivateLink service is disabled. To connect using PrivateLink again, you must re-enable the service.