23.2 23.1 22.3 22.2 22.1 21.11

Configure External Access through LoadBalancer Services

To expose your Redpanda cluster externally through load balancers, you must do the following:

  1. Create the LoadBalancer Services, one for each broker.

  2. Configure each Redpanda broker with an advertised address that points to its LoadBalancer Service.

Prerequisites

  • Install rpk on your local machine so that you can test connections to your Redpanda cluster from outside Kubernetes.

  • Understand Kubernetes networking for Redpanda.

  • Review the best practices.

  • Make sure that your Kubernetes cluster is accessible through your desired node port range. You may need to edit your inbound firewall rules.

Create the LoadBalancer Services

  1. Configure one LoadBalancer Service for each Redpanda broker in your cluster. This example manifest configures three LoadBalancer Services, one for each of the following Pods in the redpanda namespace: redpanda-0, redpanda-1, and redpanda-2. Replace the namespace and the Pod names for your deployment.

    Reveal LoadBalancer Service manifests
    redpanda-loadbalancers.yaml
    apiVersion: v1
kind: Service
metadata:
  name: lb-redpanda-0
  namespace: redpanda (1)
spec:
  type: LoadBalancer
  ports:
    - name: schemaregistry
      targetPort: 8081
      port: 8081
    - name: http
      targetPort: 8082
      port: 8082
    - name: kafka
      targetPort: 9092
      port: 9092
    - name: admin
      targetPort: 9644
      port: 9644
  selector: (2)
    statefulset.kubernetes.io/pod-name: redpanda-0
---
apiVersion: v1
kind: Service
metadata:
  name: lb-redpanda-1
  namespace: redpanda
spec:
  type: LoadBalancer
  ports:
    - name: schemaregistry
      targetPort: 8081
      port: 8081
    - name: http
      targetPort: 8082
      port: 8082
    - name: kafka
      targetPort: 9092
      port: 9092
    - name: admin
      targetPort: 9644
      port: 9644
  selector:
    statefulset.kubernetes.io/pod-name: redpanda-1
---
apiVersion: v1
kind: Service
metadata:
  name: lb-redpanda-2
  namespace: redpanda
spec:
  type: LoadBalancer
  ports:
    - name: schemaregistry
      targetPort: 8081
      port: 8081
    - name: http
      targetPort: 8082
      port: 8082
    - name: kafka
      targetPort: 9092
      port: 9092
    - name: admin
      targetPort: 9644
      port: 9644
  selector:
    statefulset.kubernetes.io/pod-name: redpanda-2
    1 Deploy the Service in the same namespace as your Redpanda brokers.
    2 Target only one Redpanda broker to be backed by the LoadBalancer Service. Every Pod in a StatefulSet is automatically given the statefulset.kubernetes.io/pod-name label, which contains the name of the Pod. By setting the spec.selector.statefulset.kubernetes.io/pod-name field of the LoadBalancer Service to the name of a Pod, it makes the LoadBalancer Service target only the individual Redpanda broker and not the whole cluster.

  2. Apply your manifest file:

    kubectl apply -f redpanda-loadbalancers.yaml

    You now have three LoadBalancer Services. Each one is linked to a specific Pod that’s running a Redpanda broker. Since each Pod should run on a dedicated worker node, each LoadBalancer Service forwards traffic from the worker node’s node port to the Redpanda broker that’s running on the worker node.

  3. Create a load balancer instance outside of your Kubernetes cluster to receive external traffic and forward it to the correct worker node:

    • Managed

    • Bare-Metal

    Your service provider should create the load balancer instances for you automatically.

    Create a load balancer instance outside of your Kubernetes cluster for each worker node and forward the traffic to the node ports that are opened by the LoadBalancer Services. You can find these node ports using this command:

    kubectl get service -n redpanda

Configure advertised addresses

You can configure brokers to advertise one of the following:

The LoadBalancer Services each open a random port from your node port range. To avoid conflicts with the node ports that are opened by the LoadBalancer Services, the following configurations disable the default NodePort Service by setting external.type to null.

Use the default DNS of the load balancer instances

If your load balancer instances have a default DNS name, you can configure each Redpanda broker to advertise the name of its corresonding load balancer instance.

  1. Find out on which worker nodes your Redpanda brokers are running.

    kubectl get pod -n redpanda  \
-o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -l \
app.kubernetes.io/component=redpanda-statefulset

  2. Find the DNS names for each of your load balancers.

  3. Add the DNS names to the external.addresses field in order. For example, the first DNS name in the list is assigned to redpanda-0, the second is assigned to redpanda-1, and so on.

    • --values

    • --set

    load-balancer-dns.yaml
    external:
  enabled: true
  type: null
  addresses:
  - <load-balancer-dns-0>
  - <load-balancer-dns-1>
  - <load-balancer-dns-2>
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values load-balancer-dns.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set external.domain=<custom-domain> \
  --set external.type=null
  --set "external.addresses={<load-balancer-dns-0>,<load-balancer-dns-1>,<load-balancer-dns-2>}"

    For default values and documentation for configuration options, see the values.yaml file.

  4. Use the load balancers' DNS names to communicate with the Redpanda cluster from outside the Kubernetes cluster:

    rpk cluster info --brokers <load-balancer-dns-name>

Use the default Redpanda subdomains

You can use a DNS provider to create CNAME records that route traffic from the default Redpanda subdomains to the addresses of the corresponding load balancer instances. The default subdomains are redpanda-<ordinal-number>, where redpanda is the name of the chart and the <ordinal-number> placeholder is generated by the StatefulSet.

  1. Get a domain from a domain registrar.

  2. Specify your domain name in the external.domain configuration.

    Replace <custom-domain> with your domain.

    • --values

    • --set

    custom-domain.yaml
    external:
  domain: <custom-domain>
    enabled: true
    type: null
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values custom-domain.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set external.enabled=true \
  --set external.type=null \
  --set external.domain=<custom-domain>

    For default values and documentation for configuration options, see the values.yaml file.

  3. Find the DNS names for each of your load balancers.

  4. Update the CNAME record for your domain so that each hostname (subdomain) points to the correct load balancer’s DNS name.

    Hostname Load balancer DNS

    redpanda-0

    <load-balancer-dns-0>

    redpanda-1

    <load-balancer-dns-1>

    redpanda-2

    <load-balancer-dns-2>

  5. Wait for your DNS changes to be propagated.

  6. Use your domain to communicate with the Redpanda cluster from outside the Kubernetes cluster:

    rpk cluster info --brokers redpanda-0.<custom-domain>

Use custom subdomains

You can give each Redpanda broker a custom subdomain to advertise instead of the default subdomains. Then, you can use a DNS provider to create CNAME records that route traffic from the custom subdomains to the corresponding load balancer instances.

  1. Get a domain from a domain registrar.

  2. Choose an option to configure the subdomains:

    • To use fully custom subdomains, do the following.

      Replace <custom-domain> with your domain, and replace the placeholders in the external.addresses configuration with your own subdomains in the order that you want them to be applied to the Redpanda brokers.

      The subdomains are given to each Redpanda broker in order. For example, the Redpanda broker running inside the redpanda-0 Pod will advertise <subdomain-for-broker-0>.<custom-domain>.

      • --values

      • --set

      custom-subdomain.yaml
      external:
  enabled: true
  type: null
  domain: <custom-domain>
  addresses:
  - <subdomain-for-broker-0>
  - <subdomain-for-broker-1>
  - <subdomain-for-broker-2>
      helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values custom-subdomain.yaml --reuse-values
      helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set external.enabled=true \
  --set external.type=null \
  --set external.domain=<custom-domain> \
  --set "external.addresses={<subdomain-for-broker0>,<subdomain-for-broker1>,<subdomain-for-broker2>}"

      For default values and documentation for configuration options, see the values.yaml file.

    • Or, to use custom subdomains that are suffixed with the index of the StatefulSet replica, do the following.

      Replace <custom-domain> with your domain, and replace <custom-subdomain> with your subdomain.

      This configuration renames your Pods to <subdomain>-<ordinal-number>. Your Redpanda brokers will advertise the <subdomain>-<ordinal-number>.<custom-domain> address.

      • --values

      • --set

      custom-subdomain-ordinal.yaml
      fullnameOverride: <custom-subdomain>
external:
  enabled: true
  type: null
  domain: <custom-domain>
      helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values custom-subdomain-ordinal.yaml --reuse-values
      helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set external.enabled=true \
  --set external.type=null \
  --set external.domain=<custom-domain> \
  --set fullnameOverride=<custom-subdomain>

      For default values and documentation for configuration options, see the values.yaml file.

  3. Find the DNS names for each of your load balancers.

  4. Update the CNAME record for your domain so that each hostname (subdomain) points to the correct load balancer’s DNS name.

    Hostname Load balancer DNS

    <subdomain-for-broker-0>

    <load-balancer-dns-0>

    <subdomain-for-broker-1>

    <load-balancer-dns-1>

    <subdomain-for-broker-2>

    <load-balancer-dns-2>

  5. Wait for your DNS changes to be propagated.

  6. Use your domain to communicate with the Redpanda cluster from outside the Kubernetes cluster:

rpk cluster info --brokers <subdomain-for-broker-0>.<custom-domain>

Next steps