23.2 23.1 22.3 22.2 22.1 21.11

Configure TLS for Redpanda in Kubernetes

By default, a Redpanda cluster communicates internally and externally over TLS (Transport Layer Security) using self-signed certificates. TLS enables secure communication between Redpanda brokers and clients.

Prerequisites

Install cert-manager.

For TLS certificate management, the Redpanda Helm chart relies on cert-manager to simplify the process of obtaining, renewing, and using certificates. cert-manager is a Kubernetes controller that defines custom Kubernetes resource types that make it easier to work with certificates. For details, see the cert-manager documentation.

Use a self-signed certificate

A self-signed certificate is signed with its own private key, instead of requesting it from a public or private certificate authority (CA).

  1. If you have disabled TLS, enable it:

    • --values

    • --set

    self-signed-tls.yaml
    tls:
  enabled: true
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values self-signed-tls.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set tls.enabled=true

    For default values and documentation for configuration options, see the values.yaml file.

  2. Make sure the Certificates are in a READY state.

    kubectl get certificate -n redpanda
    NAME                                 READY   SECRET                               AGE
redpanda-default-cert                True    redpanda-default-cert                10m
redpanda-default-root-certificate    True    redpanda-default-root-certificate    10m

Test internal connections

The SAN list of your self-signed certificate includes the internal addresses of the ClusterIP Service. As a result, you can use rpk inside the redpanda container to communicate with the cluster internally using the self-signed certificate for encryption.

Validate your internal connection to Redpanda with rpk:

kubectl exec redpanda-0 -n redpanda -c redpanda -- rpk cluster info \
  --tls-enabled \
  --brokers <broker-url>:<kafka-api-port>\
  --tls-truststore <path-to-ca-certificate>
Example output 
CLUSTER
=======
redpanda.19ae8532-c8fa-49ed-8b35-82d74813db3a

BROKERS
=======
ID    HOST                                             PORT
0*    redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
1     redpanda-1.redpanda.redpanda.svc.cluster.local.  9093
2     redpanda-2.redpanda.redpanda.svc.cluster.local.  9093

Test external connections

The SAN list of your self-signed certificate does not include the IP addresses of your worker nodes. As a result, to test external connections, you must enable external access using a custom domain. When you enable external access using a custom domain, the custom domain is included in the SAN list. Then, you can use rpk on your local machine to communicate with the cluster externally using the self-signed certificate for encryption.

  1. Configure external access to your Redpanda cluster using a custom domain.

    Your Redpanda brokers should advertise addresses in your custom domain.

  2. Install rpk on your local machine, not on a Pod:

    • Linux

    • macOS

    1. Download the rpk archive for Linux:

      curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip

    2. Ensure that you have the folder ~/.local/bin:

      mkdir -p ~/.local/bin

    3. Add it to your $PATH:

      export PATH="~/.local/bin:$PATH"

    4. Unzip the rpk files to your ~/.local/bin/ directory:

      unzip rpk-linux-amd64.zip -d ~/.local/bin/

    5. Run rpk version to display the version of the rpk binary:

      rpk version
      22.3.11 (rev 9eefb907c)

    1. If you don’t have Homebrew installed, install it.

    2. Install rpk:

      brew install redpanda-data/tap/redpanda

    3. Run rpk version to display the version of the rpk binary:

      rpk version
      22.3.11 (rev 9eefb907c)

  3. Save the root certificate authority (CA) to your local file system outside Kubernetes:

    kubectl -n redpanda get secret redpanda-default-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt

  4. Pass the root CA to rpk to validate your external connection to Redpanda.

    rpk cluster info \
  --brokers <subdomain>.<custom-domain>:<external-kafka-api-port> \
  --tls-truststore ca.crt

    Replace the following placeholders:

    • <subdomain>: The subdomain that’s in the advertised address of one of your Redpanda brokers.

    • <custom-domain>: Your domain.

    • <external-port>: The port on which your cluster is exposed.

Disable TLS

If you disable TLS, Redpanda communicates over a plain-text network connection, where any malicious party can see all communication.

To disable TLS, set tls.enabled to false:

  • --values

  • --set

no-tls.yaml
tls:
  enabled: false
helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values no-tls.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set tls.enabled=false

Troubleshooting

invalid large response size

This error appears when you don’t specify that you are connecting over TLS. For example:

kubectl exec redpanda-0 -c redpanda -n redpanda -- rpk cluster info \
  --brokers <subdomain>.<domain>:<external-kafka-api-port>

Result:

unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v1.2; is this a plaintext connection speaking to a tls endpoint?

Solution

Make sure to add the --tls-enabled option:

kubectl exec redpanda-0 -c redpanda -n redpanda -- rpk cluster info --brokers <subdomain>.<domain>:<external-kafka-api-port> --tls-enabled

i/o timeout

This error appears when your worker nodes are unreachable through the given address.

Solution

Check the following:

  • The address and port are correct.

  • Your DNS records point to addresses that resolve to your worker nodes.

Next steps

Add client authentication by combining TLS encryption with SASL authentication.