Collapse navigation tree

Collapse

Configure AWS PrivateLink in the Cloud Console

This guide is for configuring AWS PrivateLink using the Redpanda Cloud Console. To configure and manage PrivateLink on an existing public cluster, you must use the Redpanda Cloud API.

The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because these connections are treated as their own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Consider using the endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management.

  • Each client VPC can have one endpoint connected to the PrivateLink service.

  • PrivateLink allows overlapping CIDR ranges in VPC networks.

  • The number of connections is limited only by your Redpanda usage tier. PrivateLink does not add extra connection limits. However, VPC peering is limited to 125 connections. See How scalable is AWS PrivateLink?

  • You control which AWS principals are allowed to connect to the endpoint service.

Requirements

  • Your Redpanda cluster and VPC must be in the same region, unless you configure cross-region PrivateLink.

  • Use the AWS CLI to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

In Kafka clients, set connections.max.idle.ms to a value less than 350 seconds (350000 ms).

PrivateLink changes how DNS resolution works for your cluster. When you query cluster hostnames outside the VPC that contains your PrivateLink endpoint, DNS may return private IP addresses that aren’t reachable from your location.

To resolve cluster hostnames from other VPCs or on-premise networks, set up DNS forwarding using Route 53 Resolver:

  1. In the VPC that contains your PrivateLink endpoint, create a Route 53 Resolver inbound endpoint.

    Ensure that the inbound endpoint’s security group allows inbound UDP/TCP port 53 from each VPC or on-prem network that will forward queries.

  2. In each other VPC that must resolve the cluster domain, create a Resolver outbound endpoint and a forwarding rule for <cluster_domain> that targets the inbound endpoint IPs from the previous step. Associate the rule to those VPCs.

    The cluster domain is the suffix after the seed hostname. For example, if your bootstrap server URL is: seed-3da65a4a.cki01qgth38kk81ard3g.byoc.dev.cloud.redpanda.com:9092, then cluster_domain is: cki01qgth38kk81ard3g.byoc.dev.cloud.redpanda.com.

  3. For on-premises DNS, create a conditional forwarder for <cluster_domain> that forwards to the inbound endpoint IPs from the earlier step (over VPN/Direct Connect).

    Do not configure forwarding rules to target the VPC’s Amazon-provided DNS resolver (VPC base CIDR + 2). Rules must target the IP addresses of Route 53 Resolver endpoints.

Enable endpoint service for existing clusters

  1. In the Redpanda Cloud Console, select your cluster, and go to the Dataplane settings page.

  2. For AWS PrivateLink, click Enable.

  3. On the Enable PrivateLink page, for Allowed principal ARNs, click Add, and enter the Amazon Resource Names (ARNs) for each AWS principal allowed to access the endpoint service. For example, for all principals in a specific account, use arn:aws:iam::<account-id>:root. See the AWS documentation on configuring an endpoint service for details.

  4. Click Add after entering each ARN, and when finished, click Enable.

  5. (Optional) To enable cross-region PrivateLink, add supported regions. See Cross-region PrivateLink.

  6. It may take several minutes for your cluster to update. When the update is complete, the AWS PrivateLink status on the Dataplane settings page changes from In progress to Enabled.

For help with issues when enabling PrivateLink, contact Redpanda support.

When you have a PrivateLink-enabled cluster, create a VPC endpoint to connect your client VPC to your cluster.

Get cluster domain

Get the domain (cluster_domain) of the cluster from the bootstrap server URL in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

For example, if the bootstrap server URL is: seed-3da65a4a.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com:9092, then cluster_domain is: cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com.

CLUSTER_DOMAIN=<cluster_domain>
Use <cluster_domain> as the domain you target with your DNS conditional forward (optionally also *.<cluster_domain> if your DNS platform requires a wildcard).

You need the service name to create a VPC endpoint. You can find the service name on the Dataplane settings page after PrivateLink is enabled, or in the How to Connect section of the cluster overview.

PL_SERVICE_NAME=<vpc_endpoint_service_name>

With the service name stored, set up your client VPC to connect to the endpoint service.

Set up the client VPC

If you are not using an existing VPC, you must create a new one.

VPC peering and PrivateLink will not work at the same time if you set them up on the same VPC where your Kafka clients run. PrivateLink endpoints take priority.

VPC peering and PrivateLink can both be used at the same time if Kafka clients are connecting from distinct VPCs. For example, in a private Redpanda cluster, you can connect your internal Kafka clients over VPC peering, and enable PrivateLink for external services.

The client VPC must be in the same region as your Redpanda cluster, unless you have configured cross-region PrivateLink. To create the VPC, run:

# See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html for
# information on profiles and credential files
REGION=<aws-region>
PROFILE=<specific-profile-from-credential-file>

aws ec2 create-vpc --region $REGION --profile $PROFILE --cidr-block 10.0.0.0/20

# Store the client VPC ID from the command output
CLIENT_VPC_ID=<client_vpc_id>

You can also use an existing VPC. You need the VPC ID to modify its DNS attributes.

Modify VPC DNS attributes

To modify the VPC attributes, run:

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --enable-dns-hostnames "{\"Value\":true}"

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --enable-dns-support "{\"Value\":true}"

These commands enable DNS hostnames and resolution for instances in the VPC.

Create security group

You need the security group ID security_group_id from the command output to add security group rules. To create a security group, run:

aws ec2 create-security-group --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --description "Redpanda endpoint service client security group" \
    --group-name "redpanda-privatelink-sg"
SECURITY_GROUP_ID=<security_group_id>

Add security group rules

The following example adds security group rules that work for any broker count by opening the documented per-broker port ranges. For PrivateLink, clients connect to individual ports for each broker in ranges 32000-32500 (Kafka API) and 35000-35500 (HTTP Proxy). Opening only a few ports by broker count can break producers/consumers for topics with many partitions. See Private service connectivity network ports.

The following example uses 0.0.0.0/0 as the CIDR range for illustration. In production, replace 0.0.0.0/0 with the specific CIDR range of your client VPC or on-premises network to limit exposure.

 
# Allow Kafka API bootstrap (seed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 30292 --cidr 0.0.0.0/0

# Allow Schema Registry
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 30081 --cidr 0.0.0.0/0

# Allow HTTP Proxy bootstrap
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 30282 --cidr 0.0.0.0/0

# Allow Redpanda Cloud Data Plane API / Prometheus (if needed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 443 --cidr 0.0.0.0/0

# Private service connectivity broker port pools
# Kafka API per-broker ports
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID \
  --ip-permissions 'IpProtocol=tcp,FromPort=32000,ToPort=32500,IpRanges=[{CidrIp=0.0.0.0/0}]'

# HTTP Proxy per-broker ports
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID \
  --ip-permissions 'IpProtocol=tcp,FromPort=35000,ToPort=35500,IpRanges=[{CidrIp=0.0.0.0/0}]'

Create VPC subnet

You need the subnet ID subnet_id from the command output to create a VPC endpoint. Run the following command, specifying the subnet availability zone name (for example, us-west-2a):

aws ec2 create-subnet --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --availability-zone <zone> \
    --cidr-block 10.0.1.0/24
SUBNET_ID=<subnet_id>

You can also use an existing subnet from your VPC. You need the subnet ID to create a VPC endpoint.

Create VPC endpoint

Create the interface VPC endpoint using the service name and subnet ID from the previous steps:

aws ec2 create-vpc-endpoint \
    --region $REGION --profile $PROFILE \
    --vpc-id $CLIENT_VPC_ID \
    --vpc-endpoint-type "Interface" \
    --ip-address-type "ipv4" \
    --service-name $PL_SERVICE_NAME \
    --subnet-ids $SUBNET_ID \
    --security-group-ids $SECURITY_GROUP_ID \
    --private-dns-enabled

Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs for the Kafka API, Schema Registry, HTTP Proxy, and Redpanda Console are available in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

You can access Redpanda services such as Redpanda Console, Schema Registry, and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default port

Kafka API

30292

HTTP Proxy

30282

Schema Registry

30081

Redpanda Console

443

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Verify the Redpanda Console network path

When you configure private connectivity through the Cloud API, set connect_console: true on the network configuration to enable Console access through the cluster’s private endpoint. The Cloud UI sets this for you when you enable private connectivity.

The Redpanda Console URL is served on port 443 (HTTPS) and follows the form https://console-<id>.<cluster_domain>;. The <id> is a per-cluster suffix assigned by the Redpanda control plane and is opaque to you; the full URL is shown in the How to Connect section of the cluster overview in the Redpanda Cloud Console.

To verify that the network path to Redpanda Console is open, run the following commands from a host in the client network. First, confirm DNS resolves the hostname to a private IP on the cluster’s private endpoint:

dig +short console-<id>.<cluster_domain>

The response is a private IP from your client network’s address range, for example:

10.0.0.42

Then confirm Redpanda Console responds over HTTPS through the endpoint:

curl -sS -o /dev/null -w "%{http_code}\n" https://console-<id>.<cluster_domain>/

Expected output:

200

A 200 response confirms the network path. The Console UI itself does not expose a standalone login form: sign in to cloud.redpanda.com, navigate to the cluster, and use the cluster’s left navigation (Topics, Brokers, Consumer groups) to interact with Redpanda Console.

  • DNS resolution for the Redpanda Console hostname is handled automatically by the cluster’s private endpoint. You don’t need to create a private hosted zone or override DNS in the client network.

  • Ensure your network access rules (for example, AWS security groups, Azure NSGs, or GCP firewall rules) on the private endpoint allow inbound TCP on port 443 from your client workload sources only (for example, the client network’s CIDR or specific client access groups). Avoid broad source ranges such as 0.0.0.0/0.

If you are connecting from a workstation outside the client VPC, see Access Redpanda Console from a workstation.

Access Redpanda Console from a workstation

For a private BYOC cluster, Redpanda Console is exposed through the PrivateLink endpoint (or through another private route into the cluster VPC, such as VPC peering or a transit gateway), so a workstation outside the client VPC cannot resolve or reach the Console URL directly. To open Redpanda Console from a laptop, first connect the workstation to the client VPC over a VPN, then sign in to Redpanda Cloud Console and navigate to the cluster.

Common VPN options:

  • AWS Client VPN attached to the client VPC (described below).

  • A corporate VPN configured with a DNS forwarder that resolves <cluster_domain> against the client VPC’s DNS resolver.

  • An SSH SOCKS5 proxy through a bastion host in the client VPC.

The following diagram shows the network path from a workstation, through the VPN, into the client VPC, and across PrivateLink to Redpanda Console:

Workstation
  │  (public internet, OpenVPN tunnel, mutual TLS)
  ▼
AWS Client VPN endpoint        ← AWS-managed; not PrivateLink
  │  (workstation is now logically inside the client VPC)
  ▼
client VPC subnet
  │  (VPC routing; DNS resolved via VPC resolver to a private IP)
  ▼
PrivateLink VPC endpoint ENI   ← PrivateLink begins here
  │  (AWS PrivateLink service network, not on the public internet)
  ▼
Redpanda cluster VPC → Console load balancer → Redpanda Console

The client VPN tunnel itself does not use PrivateLink. The VPN’s role is to place the workstation logically inside the client VPC; the private connection to the cluster is always the PrivateLink endpoint. This is true regardless of how a client reaches the client VPC (client VPN, corporate VPN, peered network, transit gateway, SSH bastion, or an EC2 instance inside the VPC).

Set up AWS Client VPN

For full setup instructions, see the AWS Client VPN Administrator Guide. The settings specific to Redpanda PrivateLink are:

  • Authentication: mutual certificate authentication. Generate a CA, server certificate, and client certificate; upload the server certificate and the CA certificate to AWS Certificate Manager (ACM).

  • Client IPv4 CIDR: a CIDR between /22 (minimum) and /12 (maximum) that does not overlap the client VPC CIDR. For example: 10.100.0.0/22.

  • DNS server IP addresses: the IP of the client VPC’s DNS resolver (the second usable IP in the VPC CIDR). For a default VPC with CIDR 172.31.0.0/16, use 172.31.0.2. This is what makes PrivateLink hostnames resolve to the endpoint’s ENI IPs from connected clients rather than to public DNS.

  • Split-tunnel: enabled. Only traffic destined for the client VPC CIDR is routed through the VPN; the rest stays on the local internet.

  • VPC ID: the client VPC where the PrivateLink endpoint lives.

  • Target network association: associate the endpoint with a subnet in the same VPC as the PrivateLink endpoint.

  • Authorization rule: allow your client CIDR to reach the client VPC CIDR.

The client VPN endpoint takes several minutes to reach the Available state after the subnet association is created.

Allow inbound from the client VPN security group on the PrivateLink endpoint’s security group, on port 443 (Console / Schema Registry) and on the Kafka API and HTTP Proxy ports (30000-35999, covering both the seed and per-broker ports).

Connect using AWS VPN Client

  1. Install the AWS VPN Client for macOS or Windows.

  2. From the AWS Console, download the client VPN endpoint configuration file (.ovpn). Add your client certificate and private key inside <cert>…​</cert> and <key>…​</key> blocks at the bottom of the file.

  3. Open AWS VPN Client and choose File > Manage Profiles > Add Profile. Select the .ovpn file and give the profile a name.

  4. Select the profile and click Connect. The connection establishes in a few seconds.

To verify, run the following from the workstation:

dig +short console-<id>.<cluster_domain>

The response should be a private IP from the client VPC’s CIDR range. For example:

172.31.0.97

If the response shows a public address (for example, the cluster VPC’s internal LB IPs such as 10.x.x.x), the VPC’s DNS resolver was not pushed to the client. Re-check the DNS server IP addresses setting on the client VPN endpoint.

Open Redpanda Console through the Cloud Console left navigation

With the VPN connected, sign in to the Redpanda Cloud Console at cloud.redpanda.com and select your cluster.

  1. The cluster Overview page loads as normal; it is served by the Redpanda Cloud control plane and does not depend on the VPN.

  2. Click any item in the cluster’s left navigation (Topics, Brokers, Consumer groups, and so on). These views are served by Redpanda Console at https://console-<id>.<cluster_domain>;. With the VPN connected, your browser resolves the hostname to the PrivateLink endpoint’s ENI in the client VPC and the views load.

  3. If the left navigation does not load (the page hangs or returns a network error), confirm the VPN session is connected and that dig returns a private IP for the Console hostname. Without an active VPN session, only the cluster Overview page is reachable from a workstation outside the client VPC.

Redpanda Console does not provide a standalone login form. Do not open https://console-<id>.<cluster_domain>; directly in a browser: the SPA loads but cannot complete authentication, because the authentication token is handed off from Redpanda Cloud Console. Always start from cloud.redpanda.com.

Test the connection

You can test the connection to the endpoint service from any VM or container in the client VPC. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    • rpk

    • curl

    echo 'hello world' | rpk topic produce test-topic
    curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

  4. Consume from the test topic.

    • rpk

    • curl

    rpk topic consume test-topic -n 1
    curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"

By default, AWS PrivateLink only allows connections from VPCs in the same region as the endpoint service. Cross-region PrivateLink enables clients in different AWS regions to connect to your Redpanda cluster through PrivateLink.

For more information about AWS cross-region PrivateLink support, see the AWS documentation.

Prerequisites

  • The Redpanda cluster must be deployed across multiple availability zones (multi-AZ). This is an AWS limitation for cross-region PrivateLink.

Configure supported regions

The Supported regions option only appears in the UI for multi-AZ clusters.

  1. In the Redpanda Cloud Console, select your cluster, and go to the Dataplane settings page.

  2. In the AWS PrivateLink section, click Edit (or Enable if PrivateLink is not yet enabled).

  3. In the Supported regions section, click Add to add a region from which PrivateLink endpoints can connect to your cluster.

  4. Select an AWS region from the dropdown. The cluster’s home region is automatically included and not shown in the list.

  5. Repeat to add additional regions as needed.

  6. Click Save (or Enable) to apply the changes.

After saving, the Supported regions row on the Dataplane settings page displays your configured regions.

Clients in VPCs located in the supported regions can now create PrivateLink endpoints that connect to your Redpanda cluster.

Disable endpoint service

On the Dataplane settings page for the cluster, click Disable for PrivateLink. Existing connections are closed after the AWS PrivateLink service is disabled. To connect using PrivateLink again, you must re-enable the service.

Suggested reading