Use Kubernetes Secrets to manage TLS certificates

Learn how to configure Redpanda listeners with TLS using TLS certificates in your own Secret resources.

TLS, previously SSL, provides encryption for client-server communication. A server certificate prevents third parties from accessing data transferred between the client and server.

Prerequisites

You must have the following:

  • Kubernetes cluster. Ensure you have a running Kubernetes cluster, either locally, such as with minikube or kind, or remotely.

  • Kubectl. Ensure you have the kubectl command-line tool installed and configured to communicate with your cluster.

  • If you want to connect to your Redpanda cluster from outside Kubernetes, make sure to enable external access.

  • Create your TLS certificates. Make sure to include the correct addresses in the SAN:

    • If you’re creating certificates for internal listeners, make sure to include the internal addresses assigned to brokers by Kubernetes through the headless ClusterIP Service.

    • If you’re creating certificates for external listeners, make sure to include the external addresses assigned to brokers through the external.domain and/or external.addresses Helm values.

      For an example of creating the TLS certificates, see the helm-charts GitHub repository.

Create a Kubernetes Secret

A Secret is an object that contains sensitive data such as a TLS certificate and its associated key. When creating a Secret, you can specify its type using the type field of the Secret resource. The type is used to facilitate programmatic handling of the Secret data. For details, see the Kubernetes documentation.

The type of Secret you need to create depends on whether you need to store the certificate of the root certificate authority (CA) in the Secret. When using certificates issued by public certificate authorities, you don’t need to provide the root CA in the Secret. Public CAs are already trusted by default in most systems and web browsers. The trust chain is built into the operating system or web browser, which includes the root certificates of well-known CAs.

If your certificates are issued by a public CA, create a TLS Secret that includes your tls.crt and tls.key files:

kubectl create secret tls <secret-name> \
  --cert=<path>tls.crt \
  --key=<path>tls.key \
  --namespace <namespace>

If your certificates are issued by a private CA, create an Opaque Secret that includes the root CA (ca.crt) file:

kubectl create secret generic <secret-name> \
  --from-file=<path>tls.crt \
  --from-file=<path>tls.key \
  --from-file=<path>ca.crt \
  --namespace <namespace>

Replace the <path> placeholders with the paths to your certificate files.

Configure Redpanda with the Secret

You can configure Redpanda to use your TLS certificates for internal or external listeners. the default internal listeners use the TLS certificates defined in tls.certs.default. the default external listeners use the TLS certificates defined in tls.certs.external.

  1. Update your Redpanda Helm configuration to use the Secret:

    • Helm + Operator

    • Helm

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        # Internal listeners
        default:
          secretRef:
            name: <secret-name>
        # External listeners
        external:
          secretRef:
            name: <secret-name>

    If you are using a private CA, set caEnabled to true.

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        # Internal listeners
        default:
          secretRef:
            name: <secret-name>
          caEnabled: true
        # External listeners
        external:
          secretRef:
            name: <secret-name>
          caEnabled: true
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

    • --values

    • --set

    tls-secret.yml
    tls:
  enabled: true
  certs:
    # Internal listeners
    default:
      secretRef:
        name: <secret-name>
    # External listeners
    external:
      secretRef:
        name: <secret-name>

    If you are using a private CA, set caEnabled to true.

    tls-secret.yml
    tls:
  enabled: true
  certs:
    # Internal listeners
    default:
      secretRef:
        name: <secret-name>
      caEnabled: true
    # External listeners
    external:
      secretRef:
        name: <secret-name>
      caEnabled: true
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values tls-secret.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.default.secretRef.name=<secret-name>

    If you are using a private CA, set caEnabled to true.

    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.default.secretRef.name=<secret-name> \
  --set tls.certs.default.caEnabled=true

By default, certificates will be used to encrypt traffic between clients and all external listeners. You can also select specific certificates for each external listener. See Configure Listeners.

Connect to Redpanda

You can use the rpk command-line client to test both internal and external connections to Redpanda.

Test internal connections

Validate your internal connection to Redpanda with rpk by executing the following command.

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info

You should see the Kafka API endpoints for the internal listener. For example:

CLUSTER
=======
redpanda.271dac90-2dc8-48e4-9dc6-652f63684d73

BROKERS
=======
ID    HOST                                             PORT
0*    redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
1     redpanda-1.redpanda.redpanda.svc.cluster.local.  9093
2     redpanda-2.redpanda.redpanda.svc.cluster.local.  9093

Kubernetes assigns the *.redpanda.redpanda.svc.cluster.local. DNS names to the brokers through the headless ClusterIP Service. These are internal Kubernetes addresses. Port 9093 is the default port of the internal listener for the Kafka API.

Test external connections

To test external connections, external access must be enabled on your cluster and your brokers must advertise an address that’s resolvable externally by your clients.

To test external connections:

  1. Install rpk on your local machine, not inside the container:

    • Linux

    • macOS

    1. Download the rpk archive for Linux, and make sure the version matches your Redpanda version.

      • To download the latest version of rpk:

        curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip

      • To download a version other than the latest:

        curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-linux-amd64.zip

    2. Ensure that you have the folder ~/.local/bin:

      mkdir -p ~/.local/bin

    3. Add it to your $PATH:

      export PATH="~/.local/bin:$PATH"

    4. Unzip the rpk files to your ~/.local/bin/ directory:

      unzip rpk-linux-amd64.zip -d ~/.local/bin/

    5. Run rpk version to display the version of the rpk binary:

      rpk version
      23.2.21 (rev fd1019b)

    1. If you don’t have Homebrew installed, install it.

    2. Install rpk:

      brew install redpanda-data/tap/redpanda

    3. Run rpk version to display the version of the rpk binary:

      rpk version
      23.2.21 (rev fd1019b)
      This method installs the latest version of rpk, which is supported only with the latest version of Redpanda.

  2. If your TLS certificates were issued by a private CA, save the root CA to your local file system outside Kubernetes:

    kubectl --namespace <namespace> get secret <secret-name> -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt

  3. Configure rpk to connect to your cluster using the pre-configured profile:

    rpk profile create --from-profile <(kubectl get configmap --namespace <namespace> redpanda-rpk -o go-template='{{ .data.profile }}') <profile-name>

  4. Test the connection:

    rpk cluster info

You should see the Kafka API endpoints for the external listener. For example:

CLUSTER
=======
redpanda.271dac90-2dc8-48e4-9dc6-652f63684d73

BROKERS
=======
ID    HOST                                   PORT
0*    redpanda-0.customredpandadomain.local  31092
1     redpanda-1.customredpandadomain.local  31092
2     redpanda-2.customredpandadomain.local  31092

The Helm chart configures brokers with these addresses using the values of the external.domain and/or external.addresses settings in the Helm values. These are external addresses that external clients use to connect. Port 31092 is the default node port of the external listener for the Kafka API. This node port is assigned to the Kubernetes Services.

If your brokers external addresses are not resolvable, you can test external connections by sending API requests to the container port that’s assigned to the external listener. For example:

kubectl exec redpanda-0 -n redpanda -c redpanda -- rpk cluster info -X brokers=redpanda-0.redpanda.redpanda.svc.cluster.local:9094 -X tls.ca=/etc/tls/certs/external/ca.crt

Port 9094 is the default container port for the external Kafka API.

For more details on connecting to Redpanda, see Connect to Redpanda in Kubernetes.

Disable TLS

If you disable TLS, Redpanda communicates over a plain-text network connection, where any malicious party can see all communication.

To disable TLS for all listeners, set tls.enabled to false:

no-tls.yaml
tls:
  enabled: false

To disable TLS for a specific listener, set tls.enabled to false for the listener. For example, to disable TLS for the internal Kafka API listener:

listeners:
  kafka:
    tls:
      enabled: false

Troubleshoot

Here are some common troubleshooting scenarios and their solutions. For more troubleshooting steps, see Troubleshoot Redpanda in Kubernetes.

Invalid large response size

This error appears when your cluster is configured to use TLS, but you don’t specify that you are connecting over TLS.

unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v1.2; is this a plaintext connection speaking to a tls endpoint?

If you’re using rpk, ensure to add the -X tls.enabled flag, and any other necessary TLS flags such as the TLS certificate:

kubectl exec <pod-name> -c redpanda --namespace <namespace> -- rpk cluster info -X brokers=<subdomain>.<domain>:<external-port> -X tls.enabled=true

For all available flags, see the rpk command reference.

Malformed HTTP response

This error appears when a cluster has TLS enabled, and you try to access the admin API without passing the required TLS parameters.

Retrying POST for error: Post "http://127.0.0.1:9644/v1/security/users": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x03\x00\x02\x02"

If you’re using rpk, ensure to include the TLS flags.

For all available flags, see the rpk command reference.

x509: certificate signed by unknown authority

This error appears when the Certificate Authority (CA) that signed your certificates is not trusted by your system.

Check the following:

  • Ensure you have installed the root CA certificate correctly on your local system.

  • If using a self-signed certificate, ensure it is properly configured and included in your system’s trust store.

  • If you are using a certificate issued by a CA, ensure the issuing CA is included in your system’s trust store.

  • If you are using cert-manager, ensure it is correctly configured and running properly.

  • Check the validity of your certificates. They might have expired.

x509: certificate is not valid for any names

This error indicates that the certificate you are using is not valid for the specific domain or IP address you are trying to use it with. This error typically occurs when there is a mismatch between the certificate’s Subject Alternative Name (SAN) or Common Name (CN) field and the name being used to access the broker.

To resolve this issue, you may need to obtain a new certificate that is valid for the specific domain or IP address you are using. Ensure that the certificate’s SAN or CN entry matches the name being used, and that the certificate is not expired or revoked.

cannot validate certificate for 127.0.0.1

This error appears if you are using a CA certificate when you try to establish an internal connection using localhost. For example:

unable to request metadata: unable to dial: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

To resolve this issue, you must either specify the public domain or use self-signed certificates:

kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- \
  rpk cluster info \
  -X brokers=<subdomain>.<domain>:<external-port> \
  -X tls.enabled=true

I/O timeout

This error appears when your worker nodes are unreachable through the given address.

Check the following:

  • The address and port are correct.

  • Your DNS records point to addresses that resolve to your worker nodes.

Next steps

Suggested reading