Cloud
beta

Configure Dual Listener Mode

Dual listener mode is in beta and is available only on AWS. It is enabled per organization. If a request that sets connections fails with a feature-not-enabled error, contact Redpanda Support to enable dual listener mode for your organization.

Serve internet-facing clients and VPC-internal clients from the same cluster. With dual listener mode, clients inside your VPC connect over a private listener and avoid internet gateway data-processing fees, while external clients continue to connect over a public listener.

After reading this page, you will be able to:

  • Configure a cluster on AWS with both public and private listeners using Terraform or the Cloud API

  • Add the customer-managed public subnets that a BYOVPC network requires for dual listener mode

  • Migrate an existing cluster between public-only and dual listener mode

How it works

By default, a cluster is either public or private. Dual listener mode replaces that single choice with a list of connections on each cluster service: the Kafka API, HTTP Proxy, and Schema Registry. Each connection enables one listener and specifies:

  • type: Either public (served by an internet-facing load balancer) or private (served by an internal load balancer that is reachable only from inside the VPC).

  • auth: Either SASL/SCRAM or mTLS authentication.

Each connection receives its own server-assigned endpoint. For the Kafka API, the endpoint is the seed broker address. For HTTP Proxy and Schema Registry, it is the service URL. Public and private clients use different endpoints to reach the same cluster.

Configuration rules

Redpanda validates the connections configuration with the following rules, whether you configure connections through Terraform or the Cloud API:

  • If you set connections on one service, you must set it on all three services: kafka_api, http_proxy, and schema_registry.

  • All three services must use the same topology: all public, all private, or both (dual). The authentication mode can differ per service and per connection.

  • Each service accepts at most one connection per combination of type and authentication mode, so a service has at most four connections: public SASL/SCRAM, public mTLS, private SASL/SCRAM, and private mTLS.

  • You cannot combine connections with the legacy top-level connection_type field or with a per-service sasl block. When you use connections, omit both.

  • If any connection on a service uses mTLS, that service’s mtls block must set enabled to true and include at least one CA certificate in ca_certificates_pem. The public and private mTLS listeners on a service share this CA bundle. If no connection on a service uses mTLS, omit that service’s mtls block, because the API rejects it.

Prerequisites

  • A Redpanda Cloud cluster on AWS, or a plan to create one. Dual listener mode is available only on AWS. The examples on this page use a Bring Your Own Cloud (BYOC) cluster.

  • Version 2.3.0 or later of the Redpanda Terraform provider, if you use Terraform. For Bring Your Own Virtual Private Cloud (BYOVPC) clusters, also use version 2.1.16 or later of the Redpanda BYOVPC Terraform module. To set public subnets directly on the redpanda_network resource instead of through the Cloud API, use provider version 2.4.0 or later.

  • An API access token, if you use the Cloud API. See Get a Cloud API access token.

Get a Cloud API access token

  1. Save the base URL of the Redpanda Cloud API in an environment variable:

    export PUBLIC_API_ENDPOINT="https://api.cloud.redpanda.com"

  2. In the Redpanda Cloud UI, go to the Organization IAM page, and select the Service account tab. If you don’t have an existing service account, you can create a new one.

    Copy and store the client ID and secret.

    export CLOUD_CLIENT_ID=<client-id>
export CLOUD_CLIENT_SECRET=<client-secret>

  3. Get an API token using the client ID and secret. You can click the Request an API token link to see code examples to generate the token.

    export AUTH_TOKEN=`curl -s --request POST \
    --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data grant_type=client_credentials \
    --data client_id="$CLOUD_CLIENT_ID" \
    --data client_secret="$CLOUD_CLIENT_SECRET" \
    --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`

You must send the API token in the Authorization header when making requests to the Cloud API.

Create a cluster with dual listeners

Create the cluster with a connections list on each of the three services, and omit connection_type.

  • Terraform

  • Cloud API

On the redpanda_cluster resource, set connections on all three services and omit connection_type.

This example references a redpanda_resource_group and a redpanda_network defined elsewhere in your Terraform configuration.

resource "redpanda_cluster" "dual_listeners" {
  name              = "example-dual-listener-cluster"
  resource_group_id = redpanda_resource_group.example.id
  network_id        = redpanda_network.example.id
  cloud_provider    = "aws"
  region            = "us-west-2"
  cluster_type      = "byoc"
  throughput_tier   = "tier-1-aws-v3-arm"
  zones             = ["usw2-az1", "usw2-az2", "usw2-az3"]

  kafka_api = {
    connections = [
      { type = "public", auth = { mode = "sasl" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }
  http_proxy = {
    connections = [
      { type = "public", auth = { mode = "sasl" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }
  schema_registry = {
    connections = [
      { type = "public", auth = { mode = "sasl" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }
}

Each connection’s endpoint attribute is server-assigned and becomes available after apply. See Get connection endpoints.

Call POST /v1/clusters. The following example enables one public and one private SASL/SCRAM listener on each service. Set your own values for name, resource_group_id, network_id, region, zones, and throughput_tier.

CLUSTER_POST_BODY=`cat << EOF
{
    "cluster": {
        "cloud_provider": "CLOUD_PROVIDER_AWS",
        "name": "<my-dual-listener-cluster>",
        "resource_group_id": "$RESOURCE_GROUP_ID",
        "network_id": "$NETWORK_ID",
        "region": "$REGION",
        "zones": ["<zone-a>", "<zone-b>", "<zone-c>"],
        "throughput_tier": "<tier>",
        "type": "TYPE_BYOC",
        "kafka_api": {
            "connections": [
                { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
                { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
            ]
        },
        "http_proxy": {
            "connections": [
                { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
                { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
            ]
        },
        "schema_registry": {
            "connections": [
                { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
                { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
            ]
        }
    }
}
EOF`

CLUSTER_ID=`curl -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$CLUSTER_POST_BODY" $PUBLIC_API_ENDPOINT/v1/clusters | jq -r .operation.metadata.cluster_id`

The following example uses mTLS on the public listener and SASL/SCRAM on the private listener of each service. A service with an mTLS connection requires an mtls block with enabled set to true and your trusted client CA bundle in ca_certificates_pem. Send the body with the same curl command as in the previous example.

CLUSTER_POST_BODY=`cat << EOF
{
    "cluster": {
        "cloud_provider": "CLOUD_PROVIDER_AWS",
        "name": "<my-dual-listener-cluster>",
        "resource_group_id": "$RESOURCE_GROUP_ID",
        "network_id": "$NETWORK_ID",
        "region": "$REGION",
        "zones": ["<zone-a>", "<zone-b>", "<zone-c>"],
        "throughput_tier": "<tier>",
        "type": "TYPE_BYOC",
        "kafka_api": {
            "mtls": {
                "enabled": true,
                "ca_certificates_pem": ["-----BEGIN CERTIFICATE-----\n<ca-certificate>\n-----END CERTIFICATE-----\n"]
            },
            "connections": [
                { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_MTLS" } },
                { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
            ]
        },
        "http_proxy": {
            "mtls": {
                "enabled": true,
                "ca_certificates_pem": ["-----BEGIN CERTIFICATE-----\n<ca-certificate>\n-----END CERTIFICATE-----\n"]
            },
            "connections": [
                { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_MTLS" } },
                { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
            ]
        },
        "schema_registry": {
            "mtls": {
                "enabled": true,
                "ca_certificates_pem": ["-----BEGIN CERTIFICATE-----\n<ca-certificate>\n-----END CERTIFICATE-----\n"]
            },
            "connections": [
                { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_MTLS" } },
                { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
            ]
        }
    }
}
EOF`

After the cluster is ready, read the endpoint of each connection. See Get connection endpoints.

Get connection endpoints

Each connection has its own endpoint, which the server assigns when it creates the listener. Give the private endpoint to clients inside the VPC and the public endpoint to external clients.

  • Terraform

  • Cloud API

After apply, read each connection’s endpoint attribute from the connections list on the service. The following output maps each Kafka API connection to its endpoint, keyed by type and authentication mode. Replace kafka_api with http_proxy or schema_registry for the other services.

output "kafka_api_endpoints" {
  value = {
    for c in redpanda_cluster.dual_listeners.kafka_api.connections :
    "${c.type}-${c.auth.mode}" => c.endpoint
  }
}

Call GET /v1/clusters/{id}:

curl -H "Authorization: Bearer $AUTH_TOKEN" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID

In the response, each entry in a service’s connections list pairs the connection’s configuration with its endpoint. For the Kafka API, the endpoint is a seed broker address:

"kafka_api": {
  "connections": [
    {
      "config": { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
      "endpoint": "<public-seed-broker>:9092"
    },
    {
      "config": { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } },
      "endpoint": "<private-seed-broker>:9092"
    }
  ]
}

For HTTP Proxy and Schema Registry, endpoint is the service URL, including the port. The connections list is the only place in the response that reports one endpoint per listener.

To list only the type, authentication mode, and endpoint of each connection, pipe the response through jq. Replace kafka_api with http_proxy or schema_registry for the other services.

curl -s -H "Authorization: Bearer $AUTH_TOKEN" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID \
    | jq '.cluster.kafka_api.connections[] | {type: .config.type, auth: .config.auth.mode, endpoint}'

Provide public subnets for BYOVPC clusters

A dual-listener cluster places its public seed load balancer in public subnets. On a standard BYOC cluster, Redpanda manages these subnets. On a BYOVPC cluster, you provide them as customer-managed resources, with one public subnet in every availability zone that has a private (broker) subnet.

If you provision your VPC with the Redpanda BYOVPC Terraform module (version 2.1.16 or later), add one public subnet CIDR per broker availability zone to public_subnet_cidrs and set enable_public_private_connections to true. The following example shows only these inputs and the region and zones they depend on. Keep your other module inputs unchanged:

module "redpanda_byovpc" {
  source  = "redpanda-data/redpanda-byovpc/aws"
  version = ">= 2.1.16"

  region = "us-east-2"
  zones  = ["use2-az1", "use2-az2", "use2-az3"]

  # ... your existing VPC and private subnet inputs

  # One public subnet per broker AZ
  public_subnet_cidrs = [
    "10.0.16.0/20", "10.0.32.0/20", "10.0.48.0/20"
  ]
  enable_public_private_connections = true
}

enable_public_private_connections opens the public connection’s broker ports (30042-30044) to 0.0.0.0/0 on the Redpanda node security group. The private connection’s ports (30092-30094) and the Admin API stay restricted to private ranges. The module exposes the subnet ARNs through the public_subnet_arns output.

Register the module’s public subnets on the network’s customer_managed_resources.aws.public_subnets. This field is write-once. You can add it to a network that has none, but you cannot change or clear it after that. All other customer-managed resources must still be set when you create the network.

  • Terraform

  • Cloud API

In provider version 2.4.0 and later, set public_subnets directly in the redpanda_network resource’s customer_managed_resources.aws block, alongside the other customer-managed resources:

resource "redpanda_network" "example" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.example.id
  cloud_provider    = "aws"
  region            = var.region
  cluster_type      = "byoc"
  customer_managed_resources = {
    aws = {
      management_bucket = { arn = module.redpanda_byovpc.management_bucket_arn }
      dynamodb_table    = { arn = module.redpanda_byovpc.dynamodb_table_arn }
      vpc               = { arn = module.redpanda_byovpc.vpc_arn }
      private_subnets   = { arns = module.redpanda_byovpc.private_subnet_arns }
      public_subnets    = { arns = module.redpanda_byovpc.public_subnet_arns }
    }
  }
}

Apply the module, this network, and the redpanda_cluster resource (see Create a cluster with dual listeners) together in a single terraform apply. To add public subnets to a network you already created without them, add the public_subnets block to its existing configuration and apply. Terraform updates the network in place, and a cluster already on that network is unaffected. Applying a different set of subnets on a network that already has them replaces the network, the same as changing any other AWS customer-managed resource.

Before provider version 2.4.0, redpanda_network has no public_subnets field. Use the Cloud API tab instead to register the subnets, then create the redpanda_cluster resource with Terraform.

Create the network with the other customer-managed resources first (for example with Terraform, using terraform apply -target=redpanda_network.example, or with a direct POST /v1/networks call), then register the public subnets before you create the cluster:

  1. Call PATCH /v1/networks/{id}. Pass an explicit update_mask of customer_managed_resources.aws.public_subnets. Without it, the API derives the mask from the request body as the leaf path customer_managed_resources.aws.public_subnets.arns, which is not updatable, and rejects the request with INVALID_ARGUMENT.

    NETWORK_PATCH_BODY=`cat << EOF
{
    "customer_managed_resources": {
        "aws": {
            "public_subnets": {
                "arns": ["<public-subnet-arn-a>", "<public-subnet-arn-b>", "<public-subnet-arn-c>"]
            }
        }
    }
}
EOF`

curl -X PATCH \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$NETWORK_PATCH_BODY" \
    "$PUBLIC_API_ENDPOINT/v1/networks/$NETWORK_ID?update_mask=customer_managed_resources.aws.public_subnets"

    The patch runs as a long-running operation; wait for the network to return to the ready state.

  2. Create the cluster with connections on all three services (see Create a cluster with dual listeners) and the usual BYOVPC customer_managed_resources (instance profiles, security groups, cluster role, storage bucket, permissions boundary from the module outputs).

Migrate an existing cluster

The following table shows which migrations you can perform yourself with Terraform or the Cloud API, and which go through Redpanda Support:

Migration Availability

Between public-only and dual listener mode

Self-service, through Terraform or the Cloud API. Your role must include the controlplane_cluster_migrate_connectivity permission on the cluster. To grant this permission selectively, see Custom roles. Not available for BYOVPC clusters, which always keep a private connection.

Between private-only and dual listener mode

Contact Redpanda Support. All BYOVPC migrations to or from dual listener mode go through Support.

Set connections on all three services to the new topology:

  • To move from public to dual, add a private connection to each service.

  • To move from dual to public, remove the private connection from each service.

The connections list that you send replaces the service’s existing list. Any connection that you omit is removed, so include every connection that you want to keep.

If the cluster was created with the legacy connection_type field, set connections on all three services in the same request. The cluster uses connections from then on.

  • Terraform

  • Cloud API

Edit the connections list on all three services of the redpanda_cluster resource and apply. The change plans as an in-place update, and the apply waits for the migration to finish. For example, to add a private listener to each service of a public cluster:

  kafka_api = {
    connections = [
      { type = "public", auth = { mode = "sasl" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }
  http_proxy = {
    connections = [
      { type = "public", auth = { mode = "sasl" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }
  schema_registry = {
    connections = [
      { type = "public", auth = { mode = "sasl" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }

A connection added by a migration gets a new server-assigned endpoint, shown as (known after apply) in the plan. Existing connections keep their endpoints.

To migrate a cluster created with the legacy connection_type field, remove connection_type from the configuration and set connections on all three services in the same apply.

Call PATCH /v1/clusters/{id}. The following example adds a private SASL/SCRAM listener to each service of a public cluster:

CLUSTER_PATCH_BODY=`cat << EOF
{
    "kafka_api": {
        "connections": [
            { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
            { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
        ]
    },
    "http_proxy": {
        "connections": [
            { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
            { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
        ]
    },
    "schema_registry": {
        "connections": [
            { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_SASL" } },
            { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
        ]
    }
}
EOF`

curl -X PATCH \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID

The update runs as a long-running operation. Check its state with GET /v1/operations/{id}. When the operation is ready, read the endpoint of each connection. See Get connection endpoints.

Adding or removing a public connection changes the subnets that the broker nodes must occupy, and the API cannot move broker nodes between subnets. That is why migrations to or from private-only connectivity go through Redpanda Support, and why the API rejects such a request with an error that directs you to contact Support. Adding or removing a private connection only adds or removes an internal load balancer, so you can do that yourself.

Because a BYOVPC cluster must keep its private connection, the Terraform provider rejects a configuration that removes the private connection from a BYOVPC cluster at plan time with a Private Access Required error.

A cluster that uses connections cannot return to the legacy connection_type configuration. Each service must keep at least one connection.

Change the authentication mode of a connection

To switch a connection between SASL/SCRAM and mTLS, set the new authentication mode on that connection and keep every other connection in the service’s connections list. On a cluster that already uses connections, you can update one service at a time. Changing authentication modes does not change the topology, so it does not require the controlplane_cluster_migrate_connectivity permission.

The examples in this section switch the public Kafka API connection from SASL/SCRAM to mTLS and keep the private connection on SASL/SCRAM. Because a connection now uses mTLS, the service needs an mtls block with enabled set to true and your trusted client CA bundle in ca_certificates_pem.

  • Terraform

  • Cloud API

Change the connection’s auth.mode on the redpanda_cluster resource, add the service’s mtls block, and apply. The change plans as an in-place update.

  kafka_api = {
    mtls = {
      enabled             = true
      ca_certificates_pem = [file("${path.module}/client-ca.pem")]
    }
    connections = [
      { type = "public", auth = { mode = "mtls" } },
      { type = "private", auth = { mode = "sasl" } },
    ]
  }

The provider tracks each connection by its type and authentication mode, so a connection whose mode changes plans its endpoint as (known after apply) even though the listener keeps the same hostname and port.

To switch the connection back to SASL/SCRAM, set its auth.mode to sasl and remove the service’s mtls block in the same apply. The provider rejects an mtls block on a service with no mTLS connection with an Orphaned mTLS Configuration error, and rejects mtls.enabled = false on a service that still has an mTLS connection with a Conflicting mTLS Configuration error.

Call PATCH /v1/clusters/{id} with the service’s full connections list and its mtls block:

CLUSTER_PATCH_BODY=`cat << EOF
{
    "kafka_api": {
        "mtls": {
            "enabled": true,
            "ca_certificates_pem": ["-----BEGIN CERTIFICATE-----\n<ca-certificate>\n-----END CERTIFICATE-----\n"]
        },
        "connections": [
            { "type": "CONNECTION_TYPE_PUBLIC", "auth": { "mode": "AUTH_MODE_MTLS" } },
            { "type": "CONNECTION_TYPE_PRIVATE", "auth": { "mode": "AUTH_MODE_SASL" } }
        ]
    }
}
EOF`

curl -X PATCH \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID

When you switch the mode of an existing connection, the listener keeps its endpoint: the hostname and port do not change. When you add a connection with a second authentication mode on the same type, for example a public mTLS connection alongside an existing public SASL/SCRAM connection, Redpanda creates a new listener with its own endpoint. Because a switched connection keeps its port, you cannot infer the authentication mode from the port number. Read the mode and endpoint of each connection instead. See Get connection endpoints.

Limitations

  • Dual listener mode is available only on AWS.

  • Connections support SASL/SCRAM and mTLS authentication. OIDC is not supported.

  • You cannot configure dual listener mode in the Redpanda Cloud Console.

  • Migration between private-only connectivity and any configuration with a public connection requires Redpanda Support.

Suggested reading