Configure AWS PrivateLink with the Cloud API

This guide is for configuring AWS PrivateLink using the Redpanda Cloud API. To configure and manage PrivateLink on an existing public cluster, you must use the Cloud API. See Configure PrivateLink in the Cloud UI if you want to set up the endpoint service using the Redpanda Cloud UI.

The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because a PrivateLink connection is treated as its own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Consider using the PrivateLink endpoint service if you have multiple VPCs and could benefit from a more simplified approach to network management.

  • Each client VPC can have one endpoint connected to the PrivateLink service.

  • PrivateLink allows overlapping CIDR ranges in VPC networks.

  • The number of connections is limited only by your Redpanda usage tier. PrivateLink does not add extra connection limits. However, VPC peering is limited to 125 connections. See How scalable is AWS PrivateLink?

  • You control which AWS principals are allowed to connect to the endpoint service.

After getting an access token, you can enable PrivateLink when creating a new cluster, or you can enable PrivateLink for existing clusters.

Requirements

  • Install rpk.

  • Your Redpanda cluster and VPC must be in the same region.

  • In this guide, you use the Redpanda Cloud API to enable the Redpanda endpoint service for your clusters. Follow the steps below to get an access token.

  • Use the AWS CLI to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

In Kafka clients, set connections.max.idle.ms to a value less than 350 seconds.

Get a Cloud API access token

  1. Save the base URL of the Redpanda Cloud API in an environment variable:

    export PUBLIC_API_ENDPOINT="https://api.cloud.redpanda.com"

  2. In the Redpanda Cloud UI, go to the Organization IAM page, and select the Service account tab. If you don’t have an existing service account, you can create a new one.

    Copy and store the client ID and secret.

    export CLOUD_CLIENT_ID=<client-id>
export CLOUD_CLIENT_SECRET=<client-secret>

  3. Get an API token using the client ID and secret. You can click the Request an API token link to see code examples to generate the token.

    export AUTH_TOKEN=`curl -s --request POST \
    --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data grant_type=client_credentials \
    --data client_id="$CLOUD_CLIENT_ID" \
    --data client_secret="$CLOUD_CLIENT_SECRET" \
    --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`

You must send the API token in the Authorization header when making requests to the Cloud API.

  1. In the Redpanda Cloud UI, go to Resource groups and select the resource group in which you want to create a cluster.

    Copy and store the resource group ID (UUID) from the URL in the browser.

    export RESOURCE_GROUP_ID=<uuid>

  2. Call POST /v1/networks to create a network.

    Make sure to supply your own values in the following example request. The example uses a BYOC cluster. For a Dedicated cluster, set "cluster_type": "TYPE_DEDICATED". Store the network ID (network_id) after the network is created to check whether you can proceed to cluster creation.

    • name

    • cidr_block

    • aws_region

    REGION=<aws_region>

NETWORK_POST_BODY=`cat << EOF
{
    "network": {
        "cloud_provider": "CLOUD_PROVIDER_AWS",
        "cluster_type": "TYPE_BYOC",
        "name": "<my-private-link-network>",
        "cidr_block": "<10.0.0.0/20>",
        "resource_group_id": "$RESOURCE_GROUP_ID",
        "region": "$REGION"
    }
}
EOF`

NETWORK_ID=`curl -vv -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$NETWORK_POST_BODY" $PUBLIC_API_ENDPOINT/v1/networks | jq .metadata.network_id`

echo $NETWORK_ID

    Wait for the network to be ready before creating the cluster in the next step. You can check the state of the network creation by calling GET /v1/networks/{id}. You can create the cluster when the state is STATE_READY.

  3. Create a new cluster with the endpoint service enabled by calling POST /v1/clusters.

    In the example below, make sure to set your own values for the following fields:

    • zones: for example, "usw2-az1","usw2-az2","usw2-az3"

    • type: "TYPE_BYOC" or "TYPE_DEDICATED"

    • tier: for example, "tier-1-aws-v2-arm"

    • name

    • connect_console: Whether to enable connections to Redpanda Console (boolean)

    • allowed_principals: Amazon Resource Names (ARNs) for the AWS principals allowed to access the endpoint service. For example, for all principals in an account, use "arn:aws:iam::account_id:root". See Configure an endpoint service for details.

    CLUSTER_POST_BODY=`cat << EOF
{
    "cluster": {
        "cloud_provider": "CLOUD_PROVIDER_AWS",
        "connection_type": "CONNECTION_TYPE_PRIVATE",
        "name": "<my-private-link-cluster>",
        "resource_group_id": "$RESOURCE_GROUP_ID",
        "network_id": "$NETWORK_ID",
        "region": "$REGION",
        "zones": [ <zones> ],
        "throughput_tier": "<tier>",
        "type": "<type>",
        "aws_private_link": {
            "enabled": true,
            "connect_console": true,
            "allowed_principals": ["<principal_1>","<principal_2>"]
        }
    }
}
EOF`

CLUSTER_ID=`curl -vv -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$CLUSTER_POST_BODY" $PUBLIC_API_ENDPOINT/v1/clusters | jq -r .operation.metadata.cluster_id`

echo $CLUSTER_ID

    BYOC clusters only: Check that the cluster operation is completed by calling GET /v1/operations/{id}, and passing the operation ID returned from the Create Cluster call.

    When the Create Cluster operation is completed (STATE_COMPLETED), run the following rpk cloud command to finish setting up your BYOC cluster:

    rpk cloud byoc aws apply --redpanda-id=$CLUSTER_ID

Enabling PrivateLink on your VPC interrupts all communication on existing Redpanda bootstrap server and broker ports due to the change of private DNS resolution.

To avoid disruption, consider using a staged approach to enable PrivateLink. See: Switch from VPC peering to PrivateLink.

  1. In the Redpanda Cloud UI, go to the cluster overview and copy the cluster ID from the Details section.

    CLUSTER_ID=<cluster_id>

  2. Make a PATCH /v1/clusters/{cluster.id} request to update the cluster with the Redpanda Private Link Endpoint Service enabled.

    In the example below, make sure to set your own value for the following field:

    • connect_console: Whether to enable connections to Redpanda Console (boolean)

    • allowed_principals: Amazon Resource Names (ARNs) for the AWS principals allowed to access the endpoint service. For example, for all principals in an account, use "arn:aws:iam::account_id:root". See Configure an endpoint service for details.

    CLUSTER_PATCH_BODY=`cat << EOF
{
  "aws_private_link": {
    "enabled": true,
    "connect_console": true,
    "allowed_principals": ["<principal_1>","<principal_2>"]
  }
}
EOF`

curl -vv -X PATCH \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID

  3. Before proceeding, check the state of the Update Cluster operation by calling GET /v1/operations/{id}, and passing the operation ID returned from Update Cluster call. When the state is STATE_READY, proceed to the next step.

  4. Check the service state by calling GET /v1/clusters/{id}. The service_state in the aws_private_link.status response object must be Available for you to connect to the service.

    curl -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | jq '.cluster.aws_private_link.status | {service_name, service_state}'

When you have a PrivateLink-enabled cluster, you can create an endpoint to connect your VPC and your cluster.

Get cluster domain

Get the domain (cluster_domain) of the cluster from the cluster details in the Redpanda Cloud UI.

For example, if the bootstrap server URL is: seed-3da65a4a.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com:9092, then cluster_domain is: cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com.

CLUSTER_DOMAIN=<cluster_domain>

The service name is required to create VPC private endpoints. Run the following command to get the service name:

PL_SERVICE_NAME=`curl -X GET \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | jq -r .cluster.aws_private_link.status.service_name`

Create client VPC

If you are not using an existing VPC, you must create a new one.

VPC peering and PrivateLink will not work at the same time if you set them up on the same VPC where your Kafka clients run. PrivateLink endpoints take priority.

VPC peering and PrivateLink can both be used at the same time if Kafka clients are connecting from distinct VPCs. For example, in a private Redpanda cluster, you can connect your internal Kafka clients over VPC peering, and enable PrivateLink for external services.

The VPC region must be the same region where the Redpanda cluster is deployed. To create the VPC, run:

# See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html for
# information on profiles and credential files
PROFILE=<specific-profile-from-credential-file>

aws ec2 create-vpc --region $REGION --profile $PROFILE --cidr-block 10.0.0.0/20

# Store the client VPC ID from the command output
CLIENT_VPC_ID=<client_vpc_id>

You can also use an existing VPC. You need the VPC ID to modify its DNS attributes.

Modify VPC DNS attributes

To modify the VPC attributes, run:

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --enable-dns-hostnames "{\"Value\":true}"

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --enable-dns-support "{\"Value\":true}"

These commands enable DNS hostnames and resolution for instances in the VPC.

Create security group

You need the security group ID security_group_id from the command output to add security group rules. To create a security group, run:

aws ec2 create-security-group --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --description "Redpanda endpoint service client security group" \
    --group-name "${CLUSTER_ID}-sg"
SECURITY_GROUP_ID=<security_group_id>

Add security group rules

The following example adds security group rules that work for any broker count by opening the documented per-broker port ranges.

For PrivateLink, clients connect to individual ports for each broker in ranges 32000-32500 (Kafka API) and 35000-35500 (HTTP Proxy). Opening only a few ports by broker count can break producers/consumers for topics with many partitions. See Private service connectivity network ports. 
# Allow Kafka API bootstrap (seed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 30292 --cidr 0.0.0.0/0

# Allow Schema Registry
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 30081 --cidr 0.0.0.0/0

# Allow HTTP Proxy bootstrap
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 30282 --cidr 0.0.0.0/0

# Allow Redpanda Cloud Data Plane API / Prometheus (if needed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID --protocol tcp --port 443 --cidr 0.0.0.0/0

# Private service connectivity broker port pools
# Kafka API per-broker ports
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID \
  --ip-permissions 'IpProtocol=tcp,FromPort=32000,ToPort=32500,IpRanges=[{CidrIp=0.0.0.0/0}]'

# HTTP Proxy per-broker ports
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
  --group-id $SECURITY_GROUP_ID \
  --ip-permissions 'IpProtocol=tcp,FromPort=35000,ToPort=35500,IpRanges=[{CidrIp=0.0.0.0/0}]'

Create VPC subnet

You need the subnet ID subnet_id from the command output to create a VPC endpoint. Run the following command, specifying the subnet availability zone (for example, usw2-az1):

aws ec2 create-subnet --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
    --availability-zone <zone> \
    --cidr-block 10.0.1.0/24
SUBNET_ID=<subnet_id>

Create VPC endpoint

aws ec2 create-vpc-endpoint \
    --region $REGION --profile $PROFILE \
    --vpc-id $CLIENT_VPC_ID \
    --vpc-endpoint-type "Interface" \
    --ip-address-type "ipv4" \
    --service-name $PL_SERVICE_NAME \
    --subnet-ids $SUBNET_ID \
    --security-group-ids $SECURITY_GROUP_ID \
    --private-dns-enabled

Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud UI.

You can access Redpanda services such as Schema Registry and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default bootstrap port

Kafka API

30292

HTTP Proxy

30282

Schema Registry

30081

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Test the connection

You can test the PrivateLink connection from any VM or container in the consumer VPC. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    • rpk

    • curl

    echo 'hello world' | rpk topic produce test-topic
    curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

  4. Consume from the test topic.

    • rpk

    • curl

    rpk topic consume test-topic -n 1
    curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"

Suggested reading