Cloud

Configure Egress for Redpanda Connect Pipelines

Redpanda Connect pipelines on BYOC and Dedicated clusters run behind a data plane firewall that blocks outbound connections to most private addresses. Add an egress allowlist to your cluster to let pipelines reach a private destination, such as a database in a peered network or a VPC endpoint that fronts a third-party service.

After reading this page, you will be able to:

  • Identify which pipeline destinations the data plane firewall blocks by default

  • Add egress allowlist rules to a cluster using the Cloud API or Terraform

  • Troubleshoot a pipeline connection that the firewall blocks

Default pipeline egress

On BYOC and Dedicated clusters, pipelines run as pods inside the data plane VPC or VNet. Each pipeline pod enforces a firewall that classifies every outbound connection by destination address and port:

Destination Default behavior

Your Redpanda cluster

Allowed on the Kafka API, HTTP Proxy, and Schema Registry ports.

DNS resolvers inside the data plane VPC or VNet

Allowed on port 53.

Public endpoints on the internet

Allowed.

Private addresses, whether inside the data plane VPC or VNet or in a peered network

Blocked on most ports. Redpanda permits only the ports it uses to operate the cluster, and the ranges it permits can change. Add an egress allowlist entry for any private destination your pipelines need.

Link-local addresses (169.254.0.0/16), including the cloud provider instance metadata service

Blocked. You cannot allow these destinations.

The Kubernetes API server, and the kubelet ports 10250 and 10255 on the data plane CIDR

Blocked. You cannot allow these destinations.

This applies to your own networks. A database in a VPC you peer to the cluster, or anywhere in your own VPC CIDR on a BYOVPC cluster, needs an allowlist entry before pipelines can reach it.

An allowlist rule grants the pipeline permission to open the connection. It does not create a network path to the destination. If the destination sits outside the data plane VPC, you must also establish routing to it, for example through VPC peering or a transit gateway. See Networking: BYOC or Networking: Dedicated for the peering and transit gateway options available for your cluster type and cloud provider.

Prerequisites

  • A BYOC or Dedicated cluster with Redpanda Connect enabled.

  • The destination address, expressed as an IPv4 CIDR block, and the TCP or UDP port or port range it listens on.

  • To use the Cloud API: a Cloud API service account with permission to update the cluster, and an access token. See Get a Cloud API access token.

  • To use Terraform: the Redpanda Terraform provider version 2.1.1 or later.

Add allowlist rules with the Cloud API

Get a Cloud API access token

  1. Save the base URL of the Redpanda Cloud API in an environment variable:

    export PUBLIC_API_ENDPOINT="https://api.cloud.redpanda.com"

  2. In the Redpanda Cloud UI, go to the Organization IAM page, and select the Service account tab. If you don’t have an existing service account, you can create a new one.

    Copy and store the client ID and secret.

    export CLOUD_CLIENT_ID=<client-id>
export CLOUD_CLIENT_SECRET=<client-secret>

  3. Get an API token using the client ID and secret. You can click the Request an API token link to see code examples to generate the token.

    export AUTH_TOKEN=`curl -s --request POST \
    --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data grant_type=client_credentials \
    --data client_id="$CLOUD_CLIENT_ID" \
    --data client_secret="$CLOUD_CLIENT_SECRET" \
    --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`

You must send the API token in the Authorization header when making requests to the Cloud API.

Update the cluster

  1. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the Details section.

    CLUSTER_ID=<cluster_id>

  2. Make a PATCH /v1/clusters/{cluster.id} request with the destinations you want to allow. This example allows pipelines to reach a VPC endpoint on port 443, and a service listening on ports 9000 to 9100:

    CLUSTER_PATCH_BODY=`cat << EOF
{
  "redpanda_connect": {
    "allowed_destination_cidr_ports": [
      {
        "cidr": "10.0.32.0/20",
        "port_start": 443
      },
      {
        "cidr": "10.0.48.0/20",
        "port_start": 9000,
        "port_end": 9100
      }
    ]
  }
}
EOF`

curl -sS -X PATCH \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID

  3. Check the state of the Update Cluster operation by calling GET /v1/operations/{id} with the operation ID returned by the previous call. When the state is STATE_READY, the allowlist is applied.

The request replaces the full allowlist. To add a rule to an existing allowlist, call GET /v1/clusters/{id} first, then send the existing rules along with the new one.

Add allowlist rules with Terraform

Add the redpanda_connect.allowed_destination_cidr_ports attribute to your redpanda_cluster resource. Terraform updates the cluster in place without recreating it:

resource "redpanda_cluster" "example" {
  # ... other cluster arguments ...

  redpanda_connect = {
    allowed_destination_cidr_ports = [
      {
        cidr       = "10.0.32.0/20"
        port_start = 443
      },
      {
        cidr       = "10.0.48.0/20"
        port_start = 9000
        port_end   = 9100
      }
    ]
  }
}

For a pipeline example that writes to a private PostgreSQL database, see Enable egress to custom destinations.

Allowlist rule reference

Each rule allows outbound traffic to one CIDR block on one port or port range:

Field or attribute Required Description

cidr

Yes

Destination IPv4 CIDR block, for example 10.0.32.0/20. IPv6 is not supported.

port_start

Yes

First TCP or UDP port to allow, from 1 to 65535.

port_end

No

Last TCP or UDP port to allow, up to 65535. Omit this field to allow a single port. When set, the value must be greater than or equal to port_start.

Limits:

  • A cluster accepts a maximum of 16 rules.

  • Each combination of cidr, port_start, and port_end must be unique.

  • Rules cannot override the destinations that Redpanda always blocks, such as link-local addresses and the Kubernetes API server. See Default pipeline egress.

When allowlist changes take effect

A pipeline pod applies the firewall rules when it starts. When you change the allowlist, Redpanda restarts your existing pipelines so they pick up the new rules, so each pipeline stops processing briefly while it restarts. You do not need to restart pipelines yourself.

Troubleshoot a blocked connection

If a pipeline reports connection timeouts to a destination you allowed:

  1. Confirm the cluster update finished. Call GET /v1/clusters/{id} and check that redpanda_connect.allowed_destination_cidr_ports lists your rules.

  2. Confirm the pipeline restarted after the cluster update, because a pipeline picks up new rules only when it starts. See When allowlist changes take effect.

  3. Confirm the destination address falls inside the CIDR block you allowed, and that the port matches. A rule for port_start: 443 does not allow a connection to port 8443.

  4. Confirm a network path exists to the destination. An allowlist rule does not create routing. For destinations outside the data plane VPC, check your peering connection, transit gateway, or firewall rules.

  5. Confirm the destination is not one that Redpanda always blocks, such as a link-local address. See Default pipeline egress.

If the connection still fails, contact Redpanda Support.

Next steps