Collapse navigation tree

Collapse

Configure AWS PrivateLink in the Cloud Console

The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Consider using the PrivateLink endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management. You can create a new Serverless cluster with PrivateLink enabled, or enable PrivateLink for existing clusters using either the Console or the API.

  • Each client VPC can have one endpoint connected to the PrivateLink service.

  • PrivateLink allows overlapping CIDR ranges in VPC networks.

  • PrivateLink does not add extra connection limits. However, VPC peering is limited to 125 connections. See How scalable is AWS PrivateLink?

  • You control which AWS principals are allowed to connect to the endpoint service.

Requirements

  • Your Redpanda Serverless cluster and VPC must be in the same region.

  • Use the AWS CLI to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

In Kafka clients, set connections.max.idle.ms to a value less than 350 seconds.

PrivateLink changes how DNS resolution works for your cluster. When you query cluster hostnames outside the VPC that contains your PrivateLink endpoint, DNS may return private IP addresses that aren’t reachable from your location.

To resolve cluster hostnames from other VPCs or on-premise networks, set up DNS forwarding using Route 53 Resolver:

  1. In the VPC that contains your PrivateLink endpoint, create a Route 53 Resolver inbound endpoint.

    Ensure that the inbound endpoint’s security group allows inbound UDP/TCP port 53 from each VPC or on-prem network that will forward queries.

  2. In each other VPC that must resolve the cluster domain, create a Resolver outbound endpoint and a forwarding rule for <cluster_domain> that targets the inbound endpoint IPs from the previous step. Associate the rule to those VPCs.

    The cluster domain is the suffix after the seed hostname. For example, if your bootstrap server URL is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com:9092, then cluster_domain is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com.

  3. For on-premises DNS, create a conditional forwarder for <cluster_domain> that forwards to the inbound endpoint IPs from the earlier step (over VPN/Direct Connect).

    Do not configure forwarding rules to target the VPC’s Amazon-provided DNS resolver (VPC base CIDR + 2). Rules must target the IP addresses of Route 53 Resolver endpoints.

Enable endpoint service for existing clusters

If you do not already have a PrivateLink resource for your cluster’s resource group and region, create one at the organization level on the Networking page. For Serverless clusters, click Create PrivateLink.

  1. Select your cluster, and go to the Cluster settings page.

  2. Under Networking, select Private Access and then select an existing PrivateLink.

For help with issues enabling PrivateLink, contact Redpanda support.

Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

You can access Redpanda services such as the Kafka API and Schema Registry from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default bootstrap port

Kafka API

9092

Schema Registry

8081

Access Kafka API seed service

Use port 9092 to access the Kafka API seed service.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:9092'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
ID    HOST                                                                          PORT   RACK
0*    cki01qgth38kk81ard3g-0.any.us-east-1.aw.priv.prd.cloud.redpanda.com           9093   use1-az1
1     cki01qgth38kk81ard3g-1.any.us-east-1.aw.priv.prd.cloud.redpanda.com           9094   use1-az1
2     cki01qgth38kk81ard3g-2.any.us-east-1.aw.priv.prd.cloud.redpanda.com           9095   use1-az1

Access Schema Registry seed service

Use port 8081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:8081/subjects

Test the connection

You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:9092'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    echo 'hello world' | rpk topic produce test-topic

  4. Consume from the test topic.

    rpk topic consume test-topic -n 1

Disable endpoint service

On the Cluster Settings page, deselect Private Access. Existing connections are closed after the AWS PrivateLink service is disabled.

Disabling private access in Redpanda Cloud does not delete the PrivateLink endpoint in your AWS account or the PrivateLink resource in Redpanda Cloud. Both remain provisioned and continue to incur charges until you explicitly delete them.

Suggested reading