Streaming

Configure Listeners in Kubernetes

You can fine-tune client interactions with Redpanda by customizing internal and external listener configurations.

Anatomy of a listener

Listeners are gateways for client communications in Redpanda. Redpanda uses TCP sockets for client connections, where each TCP socket consists of:

  • IP address or DNS name: The unique network identifier of the Redpanda server. This can be a local address or a public address for cloud-based services.

  • Port: A specific entry point for network traffic. Redpanda listens on designated ports for client connections.

Clients must specify both the IP address or DNS name and port for a successful connection, such as 203.0.113.5:9093.

The Redpanda Helm chart defines two listeners by default:

Listener Description

default

Internal connections. These endpoints use internal addresses assigned to brokers by Kubernetes through the headless ClusterIP Service.

external

External connections. These endpoints are accessible through addresses configured in the external.domain and/or external.addresses settings in the Helm values.
Prefer external.domain for clusters that you expect to scale. Because external.addresses is a static list, extending it to add a broker forces a rolling restart of every existing broker. See Scale without restarting brokers.

For example, these are the default settings for the internal Kafka API listener:

listeners:
  kafka:
    port: 9093
    authenticationMethod:
    tls:
      enabled: true
      cert: default
      requireClientAuth: false

  • listeners.kafka.port: The container port for internal client connections.

  • listeners.kafka.authenticationMethod: The authentication method to use when the top-level auth.sasl.enabled setting is true. Default is sasl.

  • listeners.kafka.tls.enabled: Whether TLS is enabled for this listener when the top-level tls.enabled setting is true for all listeners.

  • listeners.kafka.tls.cert: The TLS certificates that this listener should use to secure client connections. TLS certificates are defined in tls.certs.

  • listeners.kafka.tls.requireClientAuth: Whether mTLS is enabled.

These are the settings for the external Kafka API listener:

listeners:
  kafka:
    external:
      default:
        enabled: true
        port: 9094
        # -- If undefined, `listeners.kafka.external.default.port` is used.
        advertisedPorts:
        - 31092
        tls:
          enabled: true
          cert: external
        # default is "sasl"
        authenticationMethod:

  • listeners.kafka.external.default.port: The container port for external client connections.

  • listeners.kafka.external.default.advertisedPorts: The node port that is routed to listeners.kafka.external.default.port. This port is opened in Kubernetes Services when external.service.enabled is true.

  • listeners.kafka.external.default.authenticationMethod: The authentication method to use when the top-level auth.sasl.enabled setting is true. Default is sasl.

  • listeners.kafka.external.default.tls.enabled: Whether TLS is enabled for this listener when the top-level tls.enabled setting is true for all listeners.

  • listeners.kafka.external.default.tls.cert: The TLS certificates that this listener should use to secure client connections. TLS certificates are defined in tls.certs.

Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API. If you configure a TLS listener for the Kafka API, you must configure listeners.schemaRegistry.tls and listeners.http.tls. All APIs, except the internal RPC port, support multiple listeners.

Scale without restarting brokers

Because the value of external.addresses is part of the checksum that determines whether Pods must restart, editing the list to add a broker rolls every existing broker. If you can’t use external.domain alone, choose one of the following patterns to keep the list unchanged when you scale.

If your addresses differ only by something each broker can work out for itself, such as its position in the cluster, set external.prefixTemplate to an expression and give external.addresses the single entry $PREFIX_TEMPLATE. Each broker evaluates the expression when it starts, so one entry serves every broker and the list never changes when you scale.

external.prefixTemplate is a shell expression, not a Helm template. It’s evaluated in the broker’s init container, where the following variables are available:

Variable Description

$POD_ORDINAL

The broker’s StatefulSet ordinal, such as 0.

$SERVICE_NAME

The broker’s Pod name, such as redpanda-0.

$KUBERNETES_NODE_NAME

The name of the worker node running the broker.

$HOST_IP_ADDRESS

The IP address of the worker node running the broker.

Command substitution works too, so you can derive a prefix rather than name it.

Replace <custom-hostname> with your hostname and <custom-domain> with your domain.

  • Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    external:
      enabled: true
      type: NodePort
      domain: <custom-domain>
      prefixTemplate: <custom-hostname>-$POD_ORDINAL
      addresses:
        - $PREFIX_TEMPLATE
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
prefix-template.yaml
external:
  enabled: true
  type: NodePort
  domain: <custom-domain>
  prefixTemplate: <custom-hostname>-$POD_ORDINAL
  addresses:
    - $PREFIX_TEMPLATE
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values prefix-template.yaml --reuse-values --wait

Each broker advertises <custom-hostname>-<pod-ordinal>.<custom-domain>. Setting external.domain also adds <custom-domain> and *.<custom-domain> to the SAN list of the chart’s TLS certificates, so this pattern works with TLS enabled.

Unlike setting fullnameOverride, this pattern doesn’t rename your Pods or Services. Use it when the hostname you advertise must differ from the name of the underlying Kubernetes resources.

The prefix is evaluated when a broker starts, not when the chart is rendered, so $PREFIX_TEMPLATE remains a literal string everywhere the chart reports addresses before startup:

  • The generated <cluster-name>-rpk ConfigMap and the NOTES output after helm upgrade show the unsubstituted string. Build client profiles from your own DNS names instead.

  • Don’t combine this pattern with external.externalDns.enabled. ExternalDNS receives the literal string as the hostname annotation and can’t create usable records. Create your DNS records yourself, such as a wildcard record for *.<custom-domain>.

external.prefixTemplate is also not part of the restart checksum. Editing it doesn’t restart your brokers, so existing brokers keep advertising their old addresses until they restart for another reason. To apply a new prefix immediately, perform a rolling restart.

Reserve addresses for brokers you haven’t added yet

If your addresses don’t follow a pattern, list an address for every broker you expect to run, not just the brokers running now. Entries beyond the current replica count are ignored, so scaling up to any broker you’ve already listed adds only the new broker and leaves the rest untouched.

statefulset:
  replicas: 3
external:
  enabled: true
  type: NodePort
  addresses:
    - <hostname-for-broker-0>
    - <hostname-for-broker-1>
    - <hostname-for-broker-2>
    # Reserved for future brokers
    - <hostname-for-broker-3>
    - <hostname-for-broker-4>
    - <hostname-for-broker-5>

Create the DNS records for the reserved addresses when you scale, not in advance.

Never set statefulset.replicas higher than the number of entries in external.addresses. The chart fails to render with an index out of range error.

Both patterns require you to know each broker’s address before the broker exists. If your addresses are assigned dynamically, such as the IP addresses of LoadBalancer Services that you don’t reserve in advance, reserve them up front if your platform supports it. Otherwise, scaling the cluster restarts every broker.

Customize the external node ports

Each listener has separate ports for internal and external connections. To customize the external node ports for each listener, replace <port> with the port that you want to use.

Redpanda doesn’t validate the configured port numbers. Make sure to verify the following:

  • Your configured port numbers are within the range that is assigned for node ports in your Kubernetes cluster.

  • Your Kubernetes cluster is accessible through your desired node port range. You may need to edit your inbound firewall rules.

  • Your configured port numbers are not in use by any other service.

For the default ports, see the Redpanda Helm Chart Specification

For example, to customize the node ports for the external Kafka API listener:

custom-kafka-port.yaml
listeners:
  kafka:
    external:
      default:
        advertisedPorts:
        - <port>

Configure TLS certificates

A Redpanda cluster provides granular control over the TLS certificates used by different listeners. This level of flexibility enables you to ensure the required level of security for each listener. For example, you can use a self-signed certificate for the internal RPC listener, while using a public certificate authority (CA) certificate for other listeners such as the Kafka API.

All listeners must use the same certificate if mTLS is enabled. Redpanda on Kubernetes does not support configuring different TLS certificates for mTLS-enabled listeners, or mixed configurations where some listeners use mTLS and others do not.

  • Internal listeners: By default, all internal listeners use the self-signed certificate defined globally in the tls.certs.default configuration. To customize the certificates for each internal listener, you can edit the listeners.<listener-name>.tls.cert setting.

  • External listeners: By default, all external listeners use the self-signed certificate defined globally in the tls.certs.external configuration. To customize the certificates for each internal listener, you can edit the listeners.<listener-name>.external.default.tls.cert setting.

Here’s an example that configures two certificates:

The external Admin API listener is configured with the public-ca-cert certificate. The internal Kafka API listener is configured with the private-ca-cert certificate, and the other listeners are configured with the default self-signed certificate.

multiple-certs-tls.yaml
tls:
  enabled: true
  certs:
    public-ca-cert:
      issuerRef:
        name: <issuer-name>
        kind: Issuer
      caEnabled: false
    private-ca-cert:
      secretRef:
        name: <secret-name>
      caEnabled: true
    default:
      caEnabled: true
listeners:
  admin:
    external:
      default:
        tls:
          cert: public-ca-cert
  kafka:
    tls:
      cert: private-ca-cert
  http:
    tls:
      cert: default
  rpc:
    tls:
      cert: default
  schemaRegistry:
    tls:
      cert: default

Disable external access

You can disable external access for all listeners or for individual listeners.

To disable external access for all listeners:

disable-external-access.yaml
external:
  enabled: false

To disable external access for a specific listener, set listeners.<listener-name>.external.default to false. For example, to disable external access to the Kafka API listener:

disable-external-kafka-api.yaml
listeners:
  kafka:
    external:
      default:
        enabled: false

Next steps

Connect to Redpanda in Kubernetes

Suggested reading