Connect to Redpanda in Kubernetes

To work with a Redpanda cluster running in Kubernetes, you can connect clients to the listeners that are exposed by Redpanda brokers. Depending on how the listeners are configured, you can access them from within the Kubernetes cluster and/or from outside the Kubernetes cluster.

API Purpose

Admin API

Operate Redpanda clusters. For example, you can modify the cluster’s configuration, decommission brokers, and place brokers in maintenance mode.

Kafka API

Interact with the Kafka protocol in Redpanda.

HTTP Proxy (PandaProxy)

Access your data through a REST API. For example, you can list topics or brokers, get events, and produce events.

Schema Registry

Store and manage event schemas. For example, you can query supported serialization formats, register schemas for a subject, and retrieve schemas of specific versions.

Prerequisites

You must have the following:

  • Kubernetes cluster: Ensure you have a running Kubernetes cluster, either locally, such as with minikube or kind, or remotely.

  • Kubectl: Ensure you have the kubectl command-line tool installed and configured to communicate with your cluster.

  • Redpanda cluster: Deploy a Redpanda cluster.

Connect to an internal cluster

To connect a client to Redpanda brokers running in the same Kubernetes cluster, use their fully qualified domain names (FQDNs) and the internal port of a listener. Together, the FQDN and internal port are called an endpoint. These endpoints may be secured using TLS and/or authentication.

The rpk client on each Redpanda broker is pre-configured to connect to the internal Admin API and internal Kafka API of the local Redpanda cluster. To use other clients, such as a Kafka client, you must configure them.

Connect internally with a local rpk client

The rpk command-line client, available on each Redpanda broker, allows you to communicate with the internal listeners of both the Admin API endpoint and the Kafka API endpoint. By default, the Redpanda Helm chart configures rpk with a local redpanda.yaml configuration file located in the /etc/redpanda/ directory. As a result, you can use rpk from inside the container. For example, this command executes the rpk cluster info command:

kubectl exec <pod-name> --namespace <namespace> -- rpk cluster info
bash

If you have SASL authentication enabled for the internal Kafka API listeners, you must specify the username, password, and SASL mechanism.

kubectl exec <pod-name> --namespace <namespace> -- rpk cluster info \
  -X user=<username>
  -X pass=<password>
  -X sasl.mechanism=<mechanism>
bash

You can also use environment variables instead of flags to specify the username, password, and SASL mechanism:

export RPK_USER=<username>
export RPK_PASS=<password>
export RPK_SASL_MECHANISM=<mechanism>
bash

For details about SASL authentication, see Configure Authentication for Redpanda in Kubernetes.

Connect to the internal Kafka API

To connect to the internal Kafka API using a client other than rpk, you must configure the client with the correct endpoints, authentication credentials, and TLS certificates.

You can find the connection details in the /etc/redpanda/redpanda.yaml file of any Pod that’s running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- cat /etc/redpanda/redpanda.yaml
bash

The rpk.kafka_api.brokers list contains the internal Kafka API endpoints of the Redpanda brokers:

redpanda.yaml
rpk:
  kafka_api:
    brokers:
      - redpanda-0.redpanda.redpanda.svc.cluster.local.:9093
      - redpanda-1.redpanda.redpanda.svc.cluster.local.:9093
      - redpanda-2.redpanda.redpanda.svc.cluster.local.:9093
yaml

If the internal listeners have SASL authentication enabled, you must also configure your clients with valid credentials. To find out if a listener has authentication enabled, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the Kafka API has SASL authentication enabled:

auth:
  sasl:
    enabled: true
listeners:
  kafka:
    port: 9093
    # default is "sasl" when empty or "null"
    authenticationMethod: null
yaml

For details about SASL authentication, see Configure Authentication for Redpanda in Kubernetes.

TLS files are stored in Secret resources that you can mount onto any Pods that run clients. TLS files may include:

  • Certificate files (*.crt): These files contain the public key and the identity (domain name) and are used for encryption. They can be self-signed or signed by a certificate authority (CA).

  • Key files (*.key): These contain the private key associated with the certificate. The private key should be kept secure and confidential.

  • CA files: These are certificates of the certificate authorities. They are used to verify if a given certificate is trusted.

You can find the names of all TLS Secrets using this command:

join -t $'\t' \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.containers[0].volumeMounts[*]}{.name}{'\t'}{.mountPath}{'\n'}{end}" | awk '$2 ~ /^\/etc\/tls\/certs\// {print $1"\t"$2}' | sort) \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.volumes[?(@.secret)]}{.name}{'\t'}{.secret.secretName}{'\n'}{end}" | sort) \
| awk 'BEGIN{printf "%-25s\t%-40s\n", "SECRET", "MOUNT PATH"} {printf "%-25s\t%-40s\n", $3, $2}'
bash

For example:

SECRET                   	MOUNT PATH
redpanda-default-cert    	/etc/tls/certs/default
redpanda-external-cert   	/etc/tls/certs/external

Then, you can mount the required Secrets into the Pods that run the clients:

apiVersion: v1
kind: Pod
metadata:
  name: redpanda-client-pod
  labels:
    app: redpanda-client
spec:
  volumes:
  - name: tls-certs
    secret:
      secretName: redpanda-default-client
  containers:
  - name: client-container
    image: example/client-image
    volumeMounts:
    - name: tls-certs
      mountPath: /etc/tls/certs
      readOnly: true
yaml

Now, you can configure clients with the mount path to the TLS files in your Secrets.

For details about TLS, see TLS for Redpanda in Kubernetes.

Connect to the internal HTTP Proxy

To connect to the HTTP Proxy, use its configured internal port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the internal port is 8082.

listeners:
  http:
    port: 8082
yaml

To test an internal connection, you can use the cURL command-line client inside the container running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8082/topics -sS
bash

If SASL authentication is enabled, provide a valid username and password using basic authentication:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8082/topics -u <username>:<password> -sS
bash

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

kubectl exec <pod-name> --namespace <namespace> -- curl https://redpanda-0.redpanda.redpanda.svc.cluster.local:8082/topics --cacert /etc/tls/certs/default/ca.crt -sS
bash
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

For all available endpoints, see HTTP Proxy API.

Connect to internal Schema Registry

To connect to the Schema Registry, use its configured internal port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the internal port is 8081.

listeners:
  schemaRegistry:
    port: 8081
yaml

To test an internal connection, you can use the cURL command-line client inside the container running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8081/subjects -sS
bash

If SASL authentication is enabled, provide a username and password using basic authentication:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8081/subjects -u <username>:<password> -sS
bash

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

kubectl exec <pod-name> --namespace <namespace> -- curl https://redpanda-0.redpanda.redpanda.svc.cluster.local:8081/subjects --cacert /etc/tls/certs/default/ca.crt -sS
bash
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

For all available endpoints, see Redpanda Schema Registry API.

Connect to the internal Admin API

To connect to the Admin API, use its configured internal port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the internal port is 8081.

listeners:
  admin:
    port: 9644
yaml

To test an internal connection, you can use the cURL command-line client inside the container running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:9644/v1/node_config -sS
bash

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

kubectl exec <pod-name> --namespace <namespace> -- curl https://redpanda-0.redpanda.redpanda.svc.cluster.local:9644/v1/node_config --cacert /etc/tls/certs/default/ca.crt -sS
bash
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

For all available endpoints, see Admin API.

Connect to an external cluster

To connect to your Redpanda cluster from outside Kubernetes, the Redpanda cluster must be configured with external access. See Configure External Access.

Create an rpk profile

An rpk profile contains a reusable configuration for a Redpanda cluster. When running rpk, you can create a profile, configure it for a cluster you’re working with, and use it repeatedly when running an rpk command for the cluster.

When external.enabled is set to true (default), the Helm chart generates a ConfigMap that contains settings for an rpk profile. You can use these settings to connect to the cluster externally.

The ConfigMap configures an rpk profile using the listeners.admin.external.default and listeners.kafka.external.default objects in Helm values.

  1. Install rpk.

  2. Configure rpk to use the profile in the ConfigMap:

    rpk profile create --from-profile <(kubectl get configmap --namespace <namespace> redpanda-rpk -o go-template='{{ .data.profile }}') <profile-name>
    bash

  3. If you have SASL authentication enabled, you must configure rpk with a valid username and password.

    When you first deploy Redpanda, the Helm chart prints some notes with the commands necessary to configure a username and password locally. For example:

    kubectl --namespace <namespace> get secret <secret-name> -o go-template="{{ range .data }}{{ . | base64decode }}{{ end }}" | IFS=: read -r RPK_USER RPK_PASS RPK_SASL_MECHANISM
export RPK_USER RPK_PASS RPK_SASL_MECHANISM
    bash

  4. If you have TLS enabled, you must save the TLS files to your local filesystem external to the Kubernetes cluster.

    When you first deploy Redpanda, the Helm chart prints some notes with the commands necessary to save the TLS files locally. For example:

    kubectl get secret --namespace <namespace> <secret-name> -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt
    bash

For more details about rpk profiles, see rpk Profiles.

Connect to the external Kafka API

To connect to the external Kafka API using a client other than rpk, you must configure the client with the correct broker endpoints, authentication credentials, and TLS certificates.

You can find the connection details in the /etc/redpanda/redpanda.yaml file of any Pod that’s running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- cat /etc/redpanda/redpanda.yaml
bash

The redpanda.advertised_kafka_api list item called default contains the external Kafka API endpoints for the Redpanda brokers:

redpanda.yaml
redpanda:
  advertised_kafka_api:
    - address: redpanda-0.redpanda.redpanda.svc.cluster.local.
      port: 9093
      name: internal
    - address: redpanda-0.customredpandadomain.local
      port: 31092
      name: default
yaml

If the external listeners have SASL authentication enabled, you must also configure your clients with valid credentials. To find out if the Redpanda cluster has authentication enabled, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the Kafka API has SASL authentication enabled:

auth:
  sasl:
    enabled: true
listeners:
  kafka:
    external:
      default:
        # default is "sasl" when empty or "null"
        authenticationMethod: null
yaml

For details about SASL authentication, see Configure Authentication for Redpanda in Kubernetes.

TLS files are stored in Secrets that you can mount onto the Pods that are running the clients. TLS files may include:

  • Certificate files (*.crt): These files contain the public key and the identity (domain name) and are used for encryption. They can be self-signed or signed by a certificate authority (CA).

  • Key files (*.key): These contain the private key associated with the certificate. The private key should be kept secure and confidential.

  • CA files: These are certificates of the certificate authorities. They are used to verify if a given certificate is trusted.

You can find the names of all TLS Secrets using this command:

join -t $'\t' \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.containers[0].volumeMounts[*]}{.name}{'\t'}{.mountPath}{'\n'}{end}" | awk '$2 ~ /^\/etc\/tls\/certs\// {print $1"\t"$2}' | sort) \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.volumes[?(@.secret)]}{.name}{'\t'}{.secret.secretName}{'\n'}{end}" | sort) \
| awk 'BEGIN{printf "%-25s\t%-40s\n", "SECRET", "MOUNT PATH"} {printf "%-25s\t%-40s\n", $3, $2}'
bash
SECRET                   	MOUNT PATH
redpanda-default-cert    	/etc/tls/certs/default
redpanda-external-cert   	/etc/tls/certs/external

Then, you can save the TLS files to your local file system. For example:

kubectl get secret --namespace <namespace> redpanda-external-cert -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt
bash

Now, you can configure clients with the path to the TLS files.

For details about TLS, see TLS for Redpanda in Kubernetes.

Connect to the external HTTP Proxy

To connect to the HTTP Proxy, use its configured external port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the external port on the container is 8082. The external node port on the worker node is 30082.

listeners:
  http:
    external:
      default:
        port: 8083
        advertisedPorts:
          - 30082
yaml

To test an external connection, you can use the cURL command-line client inside the container running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8083/topics -sS
bash

If SASL authentication is enabled, provide a username and password using basic authentication:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8083/topics -u <username>:<password> -sS
bash

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

kubectl exec <pod-name> --namespace <namespace> -- curl https://redpanda-0.redpanda.redpanda.svc.cluster.local:8083/topics --cacert /etc/tls/certs/external/ca.crt -sS
bash
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

For all available endpoints, see HTTP Proxy API.

Connect to external Schema Registry

To connect to the Schema Registry, use its configured external port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the external port on the container is 8084. The external node port on the worker node is 30081.

listeners:
  schemaRegistry:
    external:
      default:
        port: 8084
        advertisedPorts:
        - 30081
yaml

To test an external connection, you can use the cURL command-line client inside the container running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8084/subjects -sS
bash

If SASL authentication is enabled, provide a username and password using basic authentication:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:8084/subjects -u <username>:<password> -sS
bash

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

kubectl exec <pod-name> --namespace <namespace> -- curl https://redpanda-0.redpanda.redpanda.svc.cluster.local:8084/subjects --cacert /etc/tls/certs/external/ca.crt -sS
bash
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

For all available endpoints, see Redpanda Schema Registry API.

Connect to external Admin API

To connect to the Admin API, use its configured external port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all
bash

In this example, the external port on the container is 8084. The external node port on the worker node is 30081.

listeners:
  schemaRegistry:
    external:
      default:
        port: 9645
        advertisedPorts:
        - 31644
yaml

To test an external connection, you can use the cURL command-line client inside the container running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- curl http://redpanda-0.redpanda.redpanda.svc.cluster.local:9645/v1/node_config -sS
bash

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

kubectl exec <pod-name> --namespace <namespace> -- curl https://redpanda-0.redpanda.redpanda.svc.cluster.local:9645/v1/node_config --cacert /etc/tls/certs/external/ca.crt -sS
bash
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

For all available endpoints, see Admin API.

Next steps

Configure Listeners in Kubernetes

Suggested reading