Configure GCP Private Service Connect in the Cloud UI

To unlock this feature for your account, contact Redpanda support.
This guide is for configuring GCP Private Service Connect using the Redpanda Cloud UI. See Configure GCP Private Service Connect with the Cloud API if you want to set up this service using the API.

The Redpanda GCP Private Service Connect service provides secure access to Redpanda Cloud from your own VPC. Traffic over Private Service Connect does not go through the public internet because these connections are treated as their own private GCP service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Consider using the endpoint services if you have multiple VPCs and could benefit from a more simplified approach to network management:

  • Private Service Connect allows overlapping CIDR ranges in VPC networks.

  • Private Service Connect does not limit the number of connections using the service.

  • You control which GCP projects are allowed to connect to the service.

Requirements

  • Use gcloud to create the consumer-side resources, such as a client VPC and forwarding rule, or modify existing resources to use the Private Service Connect service attachment created for your cluster.

  • The client VPC must be in the same region as your Redpanda cluster.

Enable endpoint service for existing clusters

  1. In the Redpanda Cloud UI, open your cluster, and click Cluster settings.

  2. Under Private Service Connect, click Enable.

  3. For BYOC clusters with customer-managed VPC, you need a NAT subnet with the Purpose set to PRIVATE_SERVICE_CONNECT. You can create the subnet using the gcloud command-line interface (CLI):

    gcloud compute networks subnets create <subnet-name> \
    --project=<project> \
    --network=<network-name> \
    --region=<region> \
    --range=<subnet-range> \
    --purpose=PRIVATE_SERVICE_CONNECT

    Provide your values for the following placeholders:

    • <subnet-name>: The name of the NAT subnet.

    • <project>: The host GCP project ID.

    • <network-name>: The name of the VPC being used for your Redpanda Cloud cluster.

    • <region>: The region of the Redpanda Cloud cluster.

    • <subnet-range>: The CIDR range of the subnet. The mask should be at least /29. Each Private Service Connect connection takes up one IP address from the NAT subnet, so the CIDR must be able to accommodate all projects from which connections to the service attachment will be issued.

      See the GCP documentation for creating a subnet for Private Service Connect.

  4. For the accepted consumers list, you need the GCP project IDs from which incoming connections will be accepted.

  5. It may take several minutes for your cluster to update. When the update is complete, the Private Service Connect status in Cluster settings changes from In progress to Enabled.

For help with issues when enabling Private Service Connect, contact Redpanda support.

Deploy consumer-side resources

For each VPC network, you must complete the following steps to successfully connect to the service and use Kafka API and other Redpanda services such as HTTP Proxy.

  1. In Cluster settings, copy the Service attachment URL under Private Service Connect. Use this URL to create the Private Service Connect endpoint in GCP.

  2. Create a private DNS zone. Use the cluster DNS zone value as the DNS name.

    gcloud dns --project=<GCP Project ID> managed-zones create <DNS zone name> --description="<description>" --dns-name="<DNS Zone from the UI>" --visibility="private" --networks="<list of fully-qualified name of networks where the DNS zone will be visible>"

  3. In the newly-created DNS zone, create a wildcard DNS record using the cluster DNS record value.

    gcloud dns --project=rp-byoc-juan-0e38 record-sets create '*.<DNS Zone from the UI>' --zone="<DNS zone name>" --type="A" --ttl="300" --rrdatas="<PSC endpoint IP>"

  4. Confirm that your GCP VPC firewall allows traffic to and from the Private Service Connect forwarding rule IP address, on the expected ports.

Access Redpanda services through VPC endpoint

After you have enabled Private Service Connect for your cluster, your connection URLs are available in the How to Connect section of the cluster overview in the Redpanda Cloud UI.

You can access Redpanda services such as Schema Registry and HTTP Proxy from the client VPC or virtual network; for example, from a compute instance in the VPC or network.

The bootstrap server hostname is unique to each cluster. The service attachment exposes a set of bootstrap ports for access to Redpanda services. These ports load balance requests among brokers. Make sure you use the following ports for initiating a connection from a consumer:

Redpanda service Default bootstrap port

Kafka API

30292

HTTP Proxy

30282

Schema Registry

30081

Access Kafka API seed service

Use port 30292 to access the Kafka API seed service.

export REDPANDA_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
rpk cluster info -X tls.enabled=true -X user=<user> -X pass=<password>

When successful, the rpk output should look like the following:

CLUSTER
=======
redpanda.rp-cki01qgth38kk81ard3g

BROKERS
=======
ID    HOST                                                                PORT   RACK
0*    0-3da65a4a-0532364.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32092  use2-az1
1     1-3da65a4a-63b320c.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32093  use2-az1
2     2-3da65a4a-36068dc.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com  32094  use2-az1

Access Schema Registry seed service

Use port 30081 to access the Schema Registry seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.schemaregistry.v1+json" --sslv2 --http2 <schema-registry-bootstrap-server-hostname>:30081/subjects

Access HTTP Proxy seed service

Use port 30282 to access the Redpanda HTTP Proxy seed service.

curl -vv -u <user>:<password> -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 <http-proxy-bootstrap-server-hostname>:30282/topics

Test the connection

You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn’t possible right away, you can do these checks using rpk or cURL:

  1. Set the following environment variables.

    export RPK_BROKERS='<kafka-api-bootstrap-server-hostname>:30292'
export RPK_TLS_ENABLED=true
export RPK_SASL_MECHANISM="<SCRAM-SHA-256 or SCRAM-SHA-512>"
export RPK_USER=<user>
export RPK_PASS=<password>

  2. Create a test topic.

    rpk topic create test-topic

  3. Produce to the test topic.

    • rpk

    • curl

    echo 'hello world' | rpk topic produce test-topic
    curl -s \
  -X POST \
  "<http-proxy-bootstrap-server-url>/topics/test-topic" \
  -H "Content-Type: application/vnd.kafka.json.v2+json" \
  -d '{
  "records":[
      {
          "value":"hello world"
      }
  ]
}'

  4. Consume from the test topic.

    • rpk

    • curl

    rpk topic consume test-topic -n 1
    curl -s \
  "<http-proxy-bootstrap-server-url>/topics/test-topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\
  -H "Accept: application/vnd.kafka.json.v2+json"

Disable endpoint service

In Cluster settings, click Disable. Existing connections are closed after GCP Private Service Connect is disabled. To connect using Private Service Connect again, you must re-enable the service.

Suggested reading