Streaming

Connect to Redpanda in Kubernetes

To work with a Redpanda cluster running in Kubernetes, you can connect clients to the listeners that are exposed by Redpanda brokers. Depending on how the listeners are configured, you can access them from within the Kubernetes cluster and/or from outside the Kubernetes cluster.

API Purpose

Admin API

Operate Redpanda clusters. For example, you can modify the cluster’s configuration, decommission brokers, and place brokers in maintenance mode.

Kafka API

Interact with the Kafka protocol in Redpanda.

HTTP Proxy (PandaProxy)

Access your data through a REST API. For example, you can list topics or brokers, get events, and produce events.

Schema Registry

Store and manage event schemas. For example, you can query supported serialization formats, register schemas for a subject, and retrieve schemas of specific versions.

Prerequisites

You must have the following:

  • Kubernetes cluster: Ensure you have a running Kubernetes cluster, either locally, such as with minikube or kind, or remotely.

  • Kubectl: Ensure you have the kubectl command-line tool installed and configured to communicate with your cluster.

  • Redpanda cluster: Deploy a Redpanda cluster.

Connect to an internal cluster

To connect a client to Redpanda brokers running in the same Kubernetes cluster, use their fully qualified domain names (FQDNs) and the internal port of a listener. Together, the FQDN and internal port are called an endpoint. These endpoints may be secured using TLS and/or authentication.

The rpk client on each Redpanda broker is pre-configured to connect to the internal Admin API, Kafka API, and Schema Registry of the local Redpanda cluster. To use other clients, such as a Kafka client, you must configure them.

Connect internally with a local rpk client

The rpk command-line client, available on each Redpanda broker, allows you to communicate with the internal listeners of the Admin API, the Kafka API and Schema Registry. By default, the Redpanda Helm chart configures rpk with a local redpanda.yaml configuration file located in the /etc/redpanda/ directory. As a result, you can use rpk from inside the container. For example, this command executes the rpk cluster info command:

kubectl exec <pod-name> --namespace <namespace> -- rpk cluster info

If you have SASL authentication enabled for the internal Kafka API listeners, you must specify the username, password, and SASL mechanism.

kubectl exec <pod-name> --namespace <namespace> -- rpk cluster info \
  -X user=<username>
  -X pass=<password>
  -X sasl.mechanism=<mechanism>

You can also use environment variables instead of flags to specify the username, password, and SASL mechanism:

export RPK_USER=<username>
export RPK_PASS=<password>
export RPK_SASL_MECHANISM=<mechanism>

For details about SASL authentication, see Configure Authentication for Redpanda in Kubernetes.

Connect to the internal Kafka API

To connect to the internal Kafka API using a client other than rpk, you must configure the client with the correct endpoints, authentication credentials, and TLS certificates.

You can find the connection details in the /etc/redpanda/redpanda.yaml file of any Pod that’s running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- cat /etc/redpanda/redpanda.yaml

The rpk.kafka_api.brokers list contains the internal Kafka API endpoints of the Redpanda brokers:

redpanda.yaml
rpk:
  kafka_api:
    brokers:
      - <pod-name>.redpanda.<namespace>.svc.cluster.local.:9093
      - redpanda-1.redpanda.<namespace>.svc.cluster.local.:9093
      - redpanda-2.redpanda.<namespace>.svc.cluster.local.:9093

If the internal listeners have SASL authentication or TLS enabled, you must also configure your clients with valid credentials. To find out if a listener has authentication enabled, check the Helm values:

helm get values <release-name> --namespace <namespace> --all

In this example, the Kafka API has SASL authentication enabled:

auth:
  sasl:
    enabled: true
listeners:
  kafka:
    port: 9093
    # default is "sasl" when empty or "null"
    authenticationMethod: null

If the internal listeners have TLS or mTLS enabled, you must configure your clients with valid TLS files. To find out if the Redpanda cluster has TLS or mTLS enabled, check the Helm values. In this example, the Kafka API has TLS and mTLS enabled:

listeners:
  kafka:
    port: 9093
    tls:
      cert: default
      requireClientAuth: true

TLS files are stored in Secret resources that you can mount onto any Pods that run clients. TLS files may include:

  • Certificate files (*.crt): These files contain the public key and the identity (domain name) and are used for encryption. They can be self-signed or signed by a certificate authority (CA).

  • Key files (*.key): These contain the private key associated with the certificate. The private key should be kept secure and confidential.

  • CA files: These are certificates of the certificate authorities. They are used to verify if a given certificate is trusted.

You can find the names of all TLS Secrets using this command:

join -t $'\t' \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.containers[0].volumeMounts[*]}{.name}{'\t'}{.mountPath}{'\n'}{end}" | awk '$2 ~ /^\/etc\/tls\/certs\// {print $1"\t"$2}' | sort) \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.volumes[?(@.secret)]}{.name}{'\t'}{.secret.secretName}{'\n'}{end}" | sort) \
| awk 'BEGIN{printf "%-25s\t%-40s\n", "SECRET", "MOUNT PATH"} {printf "%-25s\t%-40s\n", $3, $2}'

For example:

SECRET                   	MOUNT PATH
redpanda-client          	/etc/tls/certs/redpanda-client
redpanda-default-cert    	/etc/tls/certs/default
redpanda-external-cert   	/etc/tls/certs/external

Then, you can mount the required Secrets into the Pods that run the clients:

apiVersion: v1
kind: Pod
metadata:
  name: redpanda-client-pod
  labels:
    app: redpanda-client
spec:
  volumes:
  - name: tls-certs
    secret:
      secretName: redpanda-client
  containers:
  - name: client-container
    image: example/client-image
    volumeMounts:
    - name: tls-certs
      mountPath: /etc/tls/certs
      readOnly: true

Now, you can configure clients with the mount path to the TLS files in your Secrets.

For details about TLS, see TLS for Redpanda in Kubernetes.

Connect to the internal HTTP Proxy

To connect to the HTTP Proxy, use its configured internal port.

  1. Check the Helm values to find the port:

    helm get values <release-name> --namespace <namespace> --all

    In this example, the internal port is 8082.

    listeners:
  http:
    port: 8082

  2. Use the curl command-line client inside the container running a Redpanda broker:

    kubectl exec <pod-name> --namespace <namespace> -- curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:8082/topics -sS

    If SASL authentication is enabled, provide a valid username and password using basic authentication:

    kubectl exec <pod-name> --namespace <namespace> -- curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:8082/topics -u <username>:<password> -sS

    If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

    kubectl exec <pod-name> --namespace <namespace> -- curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:8082/topics --cacert /etc/tls/certs/default/ca.crt -sS
    If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

    If mTLS is enabled, pass the path to the client’s key and certificate:

    kubectl exec <pod-name> --namespace <namespace> -- curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:8082/topics \
  --cacert /etc/tls/certs/default/ca.crt \
  --cert /etc/tls/certs/redpanda-client/tls.crt \
  --key /etc/tls/certs/redpanda-client/tls.key

For all available endpoints, see the HTTP Proxy API reference.

Connect to internal Schema Registry

  • rpk

  • curl

The rpk registry command can manage schemas directly:

kubectl exec <pod-name> --namespace <namespace> -- rpk registry subject list

To connect to the Schema Registry, use its configured internal port.

  1. Check the Helm values to find the port:

    helm get values <release-name> --namespace <namespace> --all

    In this example, the internal port is 8081.

    listeners:
  schemaRegistry:
    port: 8081

  2. Use the curl command-line client inside the container running a Redpanda broker:

    kubectl exec <pod-name> --namespace <namespace> -- curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:8081/subjects -sS

    If SASL authentication is enabled, provide a username and password using basic authentication:

    kubectl exec <pod-name> --namespace <namespace> -- curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:8081/subjects -u <username>:<password> -sS

    If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

    kubectl exec <pod-name> --namespace <namespace> -- curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:8081/subjects --cacert /etc/tls/certs/default/ca.crt -sS
    If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

    If mTLS is enabled, pass the path to the client’s key and certificate:

    kubectl exec <pod-name> --namespace <namespace> -- curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:8081/subjects \
  --cacert /etc/tls/certs/default/ca.crt \
  --cert /path/to/client.crt \
  --key /path/to/client.key

For all available endpoints, see the Schema Registry API.

Connect to the internal Admin API

  • rpk

  • curl

Using rpk, which is already configured inside the Redpanda Pod, you can interact with the Admin API. For example, to export cluster configuration:

kubectl exec <pod-name> --namespace <namespace> -- rpk cluster config export

If SASL is enabled, set your environment variables first:

export RPK_USER=<username>
export RPK_PASS=<password>
export RPK_SASL_MECHANISM=<mechanism>

If you prefer to use curl from within the Redpanda Pod:

kubectl exec <pod-name> --namespace <namespace> -- curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:9644/v1/node_config -sS

If TLS is enabled, use https and --cacert:

kubectl exec <pod-name> --namespace <namespace> -- \
  curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:9644/v1/node_config \
  --cacert /etc/tls/certs/default/ca.crt -sS

If mTLS is enabled, also include the client certificate and key:

kubectl exec <pod-name> --namespace <namespace> -- \
  curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:9644/v1/node_config \
  --cacert /etc/tls/certs/default/ca.crt \
  --cert /path/to/client.crt \
  --key /path/to/client.key -sS
If the broker’s certificate is signed by a well-known, trusted CA, and your system’s CA trust store is reliable, you may omit --cacert.

For all available endpoints, see the Admin API reference.

Connect to an external cluster

To connect to your Redpanda cluster from outside Kubernetes, the Redpanda cluster must be configured with external access. See Configure External Access.

Create an rpk profile

An rpk profile contains a reusable configuration for a Redpanda cluster. When running rpk, you can create a profile, configure it for a cluster you’re working with, and use it repeatedly when running an rpk command for the cluster.

When external.enabled is set to true (default), the Helm chart generates a ConfigMap that contains settings for an rpk profile. You can use these settings to connect to the cluster externally.

The ConfigMap configures an rpk profile using the listeners.admin.external.default and listeners.kafka.external.default objects in Helm values.

  1. Install rpk.

  2. Configure rpk to use the profile in the ConfigMap:

    rpk profile create --from-profile <(kubectl get configmap --namespace <namespace> redpanda-rpk -o go-template='{{ .data.profile }}') <profile-name>

  3. If you have SASL authentication enabled, you must configure rpk with a valid username and password.

    When you first deploy Redpanda, the Helm chart prints some notes with the commands necessary to configure a username and password locally. For example:

    kubectl --namespace <namespace> get secret <secret-name> -o go-template="{{ range .data }}{{ . | base64decode }}{{ end }}" | IFS=: read -r RPK_USER RPK_PASS RPK_SASL_MECHANISM
export RPK_USER RPK_PASS RPK_SASL_MECHANISM

  4. If you have TLS or mTLS enabled, you must save the TLS files to your local filesystem external to the Kubernetes cluster.

    When you first deploy Redpanda, the Helm chart prints some notes with the commands necessary to save the TLS files locally. For example:

    kubectl get secret --namespace <namespace> <secret-name> -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt
kubectl get secret --namespace <namespace> <secret-name> -o go-template='{{ index .data "tls.crt" | base64decode }}' > tls.crt
kubectl get secret --namespace <namespace> <secret-name> -o go-template='{{ index .data "tls.key" | base64decode }}' > tls.key

For more details about rpk profiles, see rpk Profiles.

Connect to the external Kafka API

To connect to the external Kafka API using a client other than rpk, you must configure the client with the correct broker endpoints, authentication credentials, and TLS certificates.

You can find the connection details in the /etc/redpanda/redpanda.yaml file of any Pod that’s running a Redpanda broker:

kubectl exec <pod-name> --namespace <namespace> -- cat /etc/redpanda/redpanda.yaml

The redpanda.advertised_kafka_api list item called default contains the external Kafka API endpoints for the Redpanda brokers:

redpanda.yaml
redpanda:
  advertised_kafka_api:
    - address: <pod-name>.redpanda.<namespace>.svc.cluster.local.
      port: 9093
      name: internal
    - address: <pod-name>.customredpandadomain.local
      port: 31092
      name: default

If the external listeners have SASL authentication enabled, you must also configure your clients with valid credentials. To find out if the Redpanda cluster has authentication enabled, check the Helm values:

helm get values <release-name> --namespace <namespace> --all

In this example, the Kafka API has SASL authentication enabled:

auth:
  sasl:
    enabled: true
listeners:
  kafka:
    external:
      default:
        # default is "sasl" when empty or "null"
        authenticationMethod: null

For details about SASL authentication, see Configure Authentication for Redpanda in Kubernetes.

If the external listeners have TLS or mTLS enabled, you must configure your clients with valid TLS files. To find out if the Redpanda cluster has TLS enabled, check the Helm values. In this example, the Kafka API has TLS enabled:

listeners:
  kafka:
    external:
      default:
        # enabled: true
        port: 9094
        advertisedPorts:
        - 31092
        tls:
          # enabled: true
          cert: external

TLS files are stored in Secrets that you can mount onto the Pods that are running the clients. TLS files may include:

  • Certificate files (*.crt): These files contain the public key and the identity (domain name) and are used for encryption. They can be self-signed or signed by a certificate authority (CA).

  • Key files (*.key): These contain the private key associated with the certificate. The private key should be kept secure and confidential.

  • CA files: These are certificates of the certificate authorities. They are used to verify if a given certificate is trusted.

You can find the names of all TLS Secrets using this command:

join -t $'\t' \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.containers[0].volumeMounts[*]}{.name}{'\t'}{.mountPath}{'\n'}{end}" | awk '$2 ~ /^\/etc\/tls\/certs\// {print $1"\t"$2}' | sort) \
<(kubectl get pod <pod-name> --namespace <namespace> -o jsonpath="{range .spec.volumes[?(@.secret)]}{.name}{'\t'}{.secret.secretName}{'\n'}{end}" | sort) \
| awk 'BEGIN{printf "%-25s\t%-40s\n", "SECRET", "MOUNT PATH"} {printf "%-25s\t%-40s\n", $3, $2}'
SECRET                   	MOUNT PATH
redpanda-client          	/etc/tls/certs/redpanda-client
redpanda-default-cert    	/etc/tls/certs/default
redpanda-external-cert   	/etc/tls/certs/external

Then, you can save the TLS files to your local file system. For example:

kubectl get secret --namespace <namespace> redpanda-client -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt
kubectl get secret --namespace <namespace> redpanda-client -o go-template='{{ index .data "tls.crt" | base64decode }}' > tls.crt
kubectl get secret --namespace <namespace> redpanda-client -o go-template='{{ index .data "tls.key" | base64decode }}' > tls.key

Now, you can configure clients with the path to the TLS files.

For details about TLS, see TLS for Redpanda in Kubernetes.

Connect to the external HTTP Proxy

To connect to the HTTP Proxy, use its configured external port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all

In this example, the external port on the container is 8082. The external node port on the worker node is 30082.

listeners:
  http:
    external:
      default:
        port: 8083
        advertisedPorts:
          - 30082

To test an external connection, you can use the curl command-line client inside the container running a Redpanda broker:

curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:30082/topics -sS

If SASL authentication is enabled, provide a username and password using basic authentication:

curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:30082/topics -u <username>:<password> -sS

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:30082/topics --cacert /etc/tls/certs/external/ca.crt -sS
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

If mTLS is enabled, pass the path to the client’s key and certificate:

curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:30082/topics \
  --cacert /etc/tls/certs/external/ca.crt \
  --cert /etc/tls/certs/external/tls.crt \
  --key /etc/tls/certs/external/tls.key

For all available endpoints, see the HTTP Proxy API reference.

Connect to external Schema Registry

To connect to the Schema Registry with a HTTP client, use its configured external port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all

In this example, the external port on the container is 8084. The external node port on the worker node is 30081.

listeners:
  schemaRegistry:
    external:
      default:
        port: 8084
        advertisedPorts:
        - 30081

To test an external connection, you can use the curl command-line client inside the container running a Redpanda broker:

curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:30081/subjects -sS

If SASL authentication is enabled, provide a username and password using basic authentication:

curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:30081/subjects -u <username>:<password> -sS

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:30081/subjects --cacert /etc/tls/certs/external/ca.crt -sS
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

If mTLS is enabled, pass the path to the client’s key and certificate:

curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:30081/subjects \
  --cacert /etc/tls/certs/external/ca.crt \
  --cert /etc/tls/certs/external/tls.crt \
  --key /etc/tls/certs/external/tls.key

For all available endpoints, see the Schema Registry API.

Connect to external Admin API

To connect to the Admin API using an HTTP client, use its configured external port. To find the port, check the Helm values:

helm get values <release-name> --namespace <namespace> --all

In this example, the external port on the container is 8084. The external node port on the worker node is 30081.

listeners:
  schemaRegistry:
    external:
      default:
        port: 9645
        advertisedPorts:
        - 31644

To test an external connection, you can use the curl command-line client inside the container running a Redpanda broker:

curl http://<pod-name>.redpanda.<namespace>.svc.cluster.local:31644/v1/node_config -sS

If TLS is enabled, specify the HTTPS protocol and pass the path to the ca.crt file:

curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:31644/v1/node_config --cacert /etc/tls/certs/external/ca.crt -sS
If the broker’s certificate is signed by a well-known, trusted CA, and you’re confident about the integrity of your system’s CA trust store, you don’t need the --cacert flag.

If mTLS is enabled, pass the path to the client’s key and certificate:

curl https://<pod-name>.redpanda.<namespace>.svc.cluster.local:31644/v1/node_config \
  --cacert /etc/tls/certs/external/ca.crt \
  --cert /etc/tls/certs/external/tls.crt \
  --key /etc/tls/certs/external/tls.key

For all available endpoints, see the Admin API reference.

Next steps

Configure Listeners in Kubernetes

Suggested reading