Cloud

Configure AWS PrivateLink in the Cloud Console

The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Consider using the PrivateLink endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management. You can create a new Serverless cluster with PrivateLink enabled, or enable PrivateLink for existing clusters using either the Console or the API.

  • Each client VPC can have one endpoint connected to the PrivateLink service.

  • PrivateLink allows overlapping CIDR ranges in VPC networks.

  • PrivateLink does not add extra connection limits. However, VPC peering is limited to 125 connections. See How scalable is AWS PrivateLink?

  • You control which AWS principals are allowed to connect to the endpoint service.

Requirements

  • Your Redpanda Serverless cluster and VPC must be in the same region.

  • Use the AWS CLI to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

In Kafka clients, set connections.max.idle.ms to a value less than 350 seconds (350000 ms).

PrivateLink changes how DNS resolution works for your cluster. When you query cluster hostnames outside the VPC that contains your PrivateLink endpoint, DNS may return private IP addresses that aren’t reachable from your location.

To resolve cluster hostnames from other VPCs or on-premise networks, set up DNS forwarding using Route 53 Resolver:

  1. In the VPC that contains your PrivateLink endpoint, create a Route 53 Resolver inbound endpoint.

    Ensure that the inbound endpoint’s security group allows inbound UDP/TCP port 53 from each VPC or on-prem network that will forward queries.

  2. In each other VPC that must resolve the cluster domain, create a Resolver outbound endpoint and a forwarding rule for <cluster_domain> that targets the inbound endpoint IPs from the previous step. Associate the rule to those VPCs.

    The cluster domain is the suffix after the seed hostname. For example, if your bootstrap server URL is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com:9092, then cluster_domain is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com.

  3. For on-premises DNS, create a conditional forwarder for <cluster_domain> that forwards to the inbound endpoint IPs from the earlier step (over VPN/Direct Connect).

    Do not configure forwarding rules to target the VPC’s Amazon-provided DNS resolver (VPC base CIDR + 2). Rules must target the IP addresses of Route 53 Resolver endpoints.

Enable endpoint service for existing clusters

If you do not already have a PrivateLink resource for your cluster’s resource group and region, create one at the organization level on the Networking page. For Serverless clusters, click Create PrivateLink.

  1. Select your cluster, and go to the Dataplane settings page.

  2. Under Networking, select Private Access and then select an existing PrivateLink.

For help with issues enabling PrivateLink, contact Redpanda support.

When you have a PrivateLink-enabled cluster, you can create an endpoint to connect your VPC and your cluster.

Get cluster domain

Get the domain (cluster_domain) of the cluster from the cluster details in the Redpanda Cloud Console.

For example, if the bootstrap server URL is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com:9092, then cluster_domain is: cki01qgth38kk81ard3g.any.us-east-1.aw.priv.prd.cloud.redpanda.com.

CLUSTER_DOMAIN=<cluster_domain>
Use <cluster_domain> as the domain you target with your DNS conditional forward (optionally also *.<cluster_domain> if your DNS platform requires a wildcard).

The service name is required to create VPC private endpoints. You can find the service name in the Redpanda Cloud Console on the Networking page, or by using the Redpanda Cloud API.

PL_SERVICE_NAME=<vpc_endpoint_service_name>

Create client VPC

If you are not using an existing VPC, you must create a new one.

The VPC region must be the same region where the Redpanda cluster is deployed. To create the VPC, run:

# See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html for
# information on profiles and credential files
REGION=<aws-region>
PROFILE=<specific-profile-from-credential-file>

aws ec2 create-vpc --region $REGION --profile $PROFILE --cidr-block 10.0.0.0/20

# Store the client VPC ID from the command output
CLIENT_VPC_ID=<client_vpc_id>

You can also use an existing VPC. You need the VPC ID to modify its DNS attributes.

Modify VPC DNS attributes

To modify the VPC attributes, run:

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --enable-dns-hostnames "{\"Value\":true}"

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --enable-dns-support "{\"Value\":true}"

These commands enable DNS hostnames and resolution for instances in the VPC.

Create security group

You need the security group ID security_group_id from the command output to add security group rules. To create a security group, run:

aws ec2 create-security-group --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --description "Redpanda endpoint service client security group" \
    --group-name "redpanda-privatelink-sg"
SECURITY_GROUP_ID=<security_group_id>

Add security group rules

The following example shows how to add security group rules to allow access to Redpanda services:

# Allow Kafka API bootstrap (seed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 9092 --cidr 0.0.0.0/0

# Allow Kafka API broker 1
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 9093 --cidr 0.0.0.0/0

# Allow Kafka API broker 2
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 9094 --cidr 0.0.0.0/0

# Allow Kafka API broker 3
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 9095 --cidr 0.0.0.0/0

# Allow Schema Registry
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 8081 --cidr 0.0.0.0/0

# Allow Redpanda Cloud Data Plane API / Prometheus (if needed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 443 --cidr 0.0.0.0/0

Create VPC subnet

You need the subnet ID subnet_id from the command output to create a VPC endpoint. Run the following command, specifying the subnet Availability Zone name (for example, us-west-2a):

aws ec2 create-subnet --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --availability-zone <zone> \
    --cidr-block 10.0.1.0/24
SUBNET_ID=<subnet_id>

Create VPC endpoint

The following example shows how to create the VPC endpoint:

aws ec2 create-vpc-endpoint \
    --region $REGION --profile $PROFILE \
    --vpc-id $CLIENT_VPC_ID \
    --vpc-endpoint-type "Interface" \
    --ip-address-type "ipv4" \
    --service-name $PL_SERVICE_NAME \
    --subnet-ids $SUBNET_ID \
    --security-group-ids $SECURITY_GROUP_ID \
    --private-dns-enabled

Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

You can access Redpanda services such as the Kafka API and Schema Registry from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default bootstrap port

Kafka API

9092

Schema Registry

8081

Access Kafka API seed service

Use port 9092 to access the Kafka API seed service.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:9092'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
ID    HOST                                                                          PORT   RACK
0*    cki01qgth38kk81ard3g-0.any.us-east-1.aw.priv.prd.cloud.redpanda.com           9093   use1-az1
1     cki01qgth38kk81ard3g-1.any.us-east-1.aw.priv.prd.cloud.redpanda.com           9094   use1-az1
2     cki01qgth38kk81ard3g-2.any.us-east-1.aw.priv.prd.cloud.redpanda.com           9095   use1-az1

Access Schema Registry seed service

Use port 8081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:8081/subjects

Test the connection

You can test the connection to the endpoint service from any VM or container in the client VPC. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:9092'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    echo 'hello world' | rpk topic produce test-topic

  4. Consume from the test topic.

    rpk topic consume test-topic -n 1

Disable endpoint service

On the Dataplane settings page, deselect Private Access. Existing connections are closed after the AWS PrivateLink service is disabled.

Disabling private access in Redpanda Cloud does not delete the PrivateLink endpoint in your AWS account or the PrivateLink resource in Redpanda Cloud. Both remain provisioned and continue to incur charges until you explicitly delete them.

Suggested reading