Configuring the Kubernetes Operator for Connectivity

Deprecated red

The Cluster and Console resources are deprecated. For details, see the deprecation notice. To migrate to the Redpanda resource, see Migrate from Cluster and Console Custom Resources

The Redpanda operator supports configuration for internal and external connectivity. This section covers the corresponding configuration settings, the main actions the operator takes, the resources created, and the expected outputs.

Internal connectivity

This section uses a modified version of the one node cluster example with three nodes.

Custom Resource

The Cluster Custom Resource (CR) lets you configure a listener for each API, with each listener containing the desired port number. For example:

  configuration:
    kafkaApi:
    - port: 9092
    pandaproxyApi:
    - port: 8082
    schemaRegistry:
      port: 8081
    adminApi:
    - port: 9644

Created service

This example creates a single, headless Kubernetes Service (with clusterIP set to None), since a Kafka client must have direct access to individual brokers instead of load balancing requests across the brokers. The service has a port for each listener:

$ kubectl describe svc three-node-cluster
Name:              three-node-cluster
..
Type:              ClusterIP
IP Families:       <none>
IP:                None
IPs:               <none>
Port:              admin  9644/TCP
TargetPort:        9644/TCP
Endpoints:         10.0.1.7:9644, ..
Port:              kafka  9092/TCP
TargetPort:        9092/TCP
Endpoints:         10.0.1.7:9092, ..
Port:              proxy  8082/TCP
TargetPort:        8082/TCP
Endpoints:         10.0.1.7:8082, ..

Each listener is given three fields. In this example, the Kafka listener has a named port kafka with port 9092, as specified in the CR. It’s also given an identical target port that clients in the internal network can use for communication, and the listener gets three endpoints with the internal IP addresses of the service.

For more information on services, see the Kubernetes documentation.

Redpanda configuration

In addition to creating the headless service, the Redpanda operator prepares a redpanda.yaml configuration file. The configuration is read during startup and contains the advertised addresses. In this case, they’re set to the FQDN of each broker:

$ kubectl logs three-node-cluster-0
..
advertised_kafka_api:
    - address: three-node-cluster-0.three-node-cluster.default.svc.cluster.local.
      name: kafka
      port: 9092
  ...
  kafka_api:
    - address: 0.0.0.0
      name: kafka
      port: 9092

Cluster status

Similarly, the CR status contains the advertised internal addresses:

$ kubectl describe cluster three-node-cluster
..
Status:
  Nodes:
    Internal:
      three-node-cluster-0.three-node-cluster.default.svc.cluster.local.
      three-node-cluster-1.three-node-cluster.default.svc.cluster.local.
      three-node-cluster-2.three-node-cluster.default.svc.cluster.local.
    Schema Registry:
      Internal:  three-node-cluster-cluster.default.svc.cluster.local:8081

Try it out

A client in the internal network can use the following addresses to communicate with the brokers using the Kafka API:

BROKERS=`kubectl get clusters three-node-cluster -o=jsonpath='{.status.nodes.internal}'  | jq -r 'join(",")'`

The result should show this FQDN list:

$ echo $BROKERS
three-node-cluster-0.three-node-cluster.default.svc.cluster.local.,\
three-node-cluster-1.three-node-cluster.default.svc.cluster.local.,\
three-node-cluster-2.three-node-cluster.default.svc.cluster.local.

To check the cluster info:

rpk -X brokers=$BROKERS cluster info

This example focuses on the Kafka API. Similar information is available for the Admin API and the HTTP Proxy API. Regardless of the API, the internal addresses can only be used by clients within the internal network.

External connectivity

This section uses the external connectivity specification as a base.

Custom Resource

To create a cluster that is externally accessible, you can enable external connectivity by adding a second listener for each API through the external field in the CR. In particular, you can use TLS or mTLS on the external port and keep the internal port open. To enable external connectivity for all supported APIs:

  configuration:
    kafkaApi:
     - port: 9092
     - external:
         enabled: true
    pandaproxyApi:
     - port: 8082
     - external:
         enabled: true
    schemaRegistry:
      port: 8081
      external:
        enabled: true
    adminApi:
    - port: 9644
     - external:
         enabled: true

Created services

You now have two listeners: one internal and one external. The operator creates the headless service (as in the internal connectivity case), and it also creates two services to support external connectivity:

  1. A load-balanced ClusterIP service used as an entry point for HTTP Proxy.

  2. A Nodeport service used to expose each API to the node’s external network.

<cluster-name>
headless

<cluster-name>-cluster
load-balanced

<cluster-name>-external
nodeports

Admin API

y

n

y

Kafka API

y

n

y

HTTP Proxy API

y

y

y

Schema Registry

y

y

y

Each external listener is provided with a NodePort that is automatically selected by Kubernetes. A benefit of not specifying one explicitly is the prevention of port collisions. Each NodePort has a corresponding internal port, which is set by convention to the internal-port +1. As a result, port 9092 becomes 9093, and the port name has -external appended to it.

$ kubectl describe svc external-connectivity-external
...
Port:                     kafka-external  9093/TCP
TargetPort:               9093/TCP
NodePort:                 kafka-external  31848/TCP
Endpoints:                <none>

Make sure that the generated NodePorts are open and reachable.

ClusterIP service

The ClusterIP service load-balances requests by Kubernetes to the Redpanda nodes with the help of a selector. The following service description shows a three-node Redpanda cluster with three endpoints. The -cluster service is currently used by the HTTP Proxy API as a bootstrap point.

$ kubectl describe svc external-connectivity-cluster

Name:              external-connectivity-cluster
..
Selector:          ..., app.kubernetes.io/name=redpanda
Type:              ClusterIP
IP Families:       <none>
IP:                10.3.246.143
IPs:               <none>
Port:              proxy-external  8083/TCP
TargetPort:        8083/TCP
Endpoints:         10.0.0.8:8083,10.0.1.8:8083,10.0.2.4:8083

External service

The -external service is responsible for setting up NodePorts for each API. Because external connectivity is enabled for all three APIs, you have three NodePorts, each one pointing to a target port with a port number of the original port + 1.

$ kubectl describe svc external-connectivity-external
Name:                     external-connectivity-external
..
Selector:                 <none>
Type:                     NodePort
IP Families:              <none>
IP:                       10.3.247.127
IPs:                      <none>

Port:                     kafka-external  9093/TCP
TargetPort:               9093/TCP
NodePort:                 kafka-external  31848/TCP
Endpoints:                <none>

Port:                     admin-external  9645/TCP
TargetPort:               9645/TCP
NodePort:                 admin-external  31490/TCP
Endpoints:                <none>

Port:                     proxy-external  8083/TCP
TargetPort:               8083/TCP
NodePort:                 proxy-external  30638/TCP
Endpoints:                <none>

Assuming a client has network access to the node and the NodePorts are open, a client can use the following addresses to communicate with the brokers using the configured APIs:

BROKERS=`kubectl get clusters external-connectivity -o=jsonpath='{.status.nodes.external}'  | jq -r 'join(",")'`

The result contains a list of external IPs with the broker port:

$ echo $BROKERS
<node-0-external-ip>:31848,<node-1-external-ip>:31848,<node-2-external-ip>:31848

Given the list of addresses, you can use rpk (or a Kafka client) to test the connection:

rpk -X brokers=$BROKERS cluster info

You can use the same steps to configure external access for the Admin API and the HTTP Proxy API. Remember that the internal addresses can only be used by clients within the internal network, so to allow external connections, the node must have a reachable external IP, and the NodePorts must be open to the client.

Redpanda configuration

The Redpanda operator prepares the redpanda.yaml configuration file. For each API, the configuration file contains an additional external listener and its corresponding advertised address. The second advertised address points to the external IP of each node and the NodePort of that API. For example:

$ kubectl exec external-connectivity -- cat /etc/redpanda/redpanda.yaml
  advertised_kafka_api:
    - address: external-connectivity-0.external-connectivity.default.svc.cluster.local.
      name: kafka
      port: 9092
    - address: <external-node-ip>
      name: kafka-external
      port: 31848

Container configuration

The Redpanda operator ensures that the Redpanda container exposes the requested ports, as described in the CR. The Redpanda containers are configured to expose two ports per API: one internal and one external. The operator does not create an external port for RPC.

By running kubectl describe pod external-connectivity-0, you can see the ports and the mapping to the node ports created through the NodePort service:

$ kubectl describe pod external-connectivity-0
...
Containers:
  redpanda:
    ..
    Ports:         33145/TCP, 9644/TCP, 9092/TCP, 8082/TCP, 9093/TCP, 9645/TCP, 8083/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 31848/TCP, 31490/TCP, 30638/TCP

Port summary: configuration based on example specification

Internal listener ports External listener ports (port+1:nodeport)

Admin API

9644

9645:31490

Kafka API

9092

9093:31848

HTTP Proxy API

8082

8083:30638

Using names instead of external IPs

The CRD includes a subdomain field that lets you specify the advertised address of external listeners. For example, for the Kafka API:

  configuration:
    kafkaApi:
    - port: 9092
    - external:
        enabled: true
        subdomain: "test.subdomain.com"

The generated redpanda.yaml configuration uses the subdomain field to generate the advertised addresses for the external listeners following this format: \<broker_id\>.\<subdomain\>:\<node_port\>.

DNS configuration is not handled by the Redpanda operator.

The Redpanda configuration reflects this in the advertised addresses:

$ kubectl exec external-connectivity -- cat /etc/redpanda/redpanda.yaml
...
redpanda:
  advertised_kafka_api:
    - address: external-connectivity-0.external-connectivity.default.svc.cluster.local.
      name: kafka
      port: 9092
    - address: 0.test.subdomain.com
      name: kafka-external
      port: 31631

The CR contains the addresses in its status:

$ kubectl describe cluster external-connectivity
...
Status:
  Nodes:
    External:
      0.test.subdomain.com:31631
      1.test.subdomain.com:31631
      2.test.subdomain.com:31631

Suggested reading