Configure TLS for Redpanda in Kubernetes

By default, a Redpanda cluster communicates internally and externally over TLS (Transport Layer Security) using self-signed certificates. TLS enables secure communication between Redpanda brokers and clients.

Prerequisites

Install cert-manager.

For TLS certificate management, the Redpanda Helm chart relies on cert-manager to simplify the process of obtaining, renewing, and using certificates. cert-manager is a Kubernetes controller that defines custom Kubernetes resource types that make it easier to work with certificates. For details, see the cert-manager documentation.

Use a self-signed certificate

A self-signed certificate is signed with its own private key, instead of requesting it from a public or private certificate authority (CA).

1. If you have disabled TLS, enable it:

   --values

   self-signed-tls.yaml

   tls:
     enabled: true

   helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
     --values self-signed-tls.yaml --reuse-values

   --set

   helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
     --set tls.enabled=true
   note

   For default values and documentation for configuration options, see the values.yaml file.

2. Make sure the Certificates are in a READY state.

   kubectl get certificate -n redpanda

   NAME                                 READY   SECRET                               AGE
   redpanda-default-cert                True    redpanda-default-cert                10m
   redpanda-default-root-certificate    True    redpanda-default-root-certificate    10m

Test internal connections

The SAN list of your self-signed certificate includes the internal addresses of the ClusterIP Service. As a result, you can use rpk inside the redpanda container to communicate with the cluster internally using the self-signed certificate for encryption.

Validate your internal connection to Redpanda with rpk:

kubectl exec redpanda-0 -n redpanda -c redpanda -- rpk cluster info \
  --tls-enabled \
  --brokers <broker-url>:<kafka-api-port>\
  --tls-truststore <path-to-ca-certificate>

Example output

CLUSTER
=======
redpanda.19ae8532-c8fa-49ed-8b35-82d74813db3a

BROKERS
=======
ID    HOST                                             PORT
0*    redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
1     redpanda-1.redpanda.redpanda.svc.cluster.local.  9093
2     redpanda-2.redpanda.redpanda.svc.cluster.local.  9093

Test external connections

The SAN list of your self-signed certificate does not include the IP addresses of your worker nodes. As a result, to test external connections, you must enable external access using a custom domain. When you enable external access using a custom domain, the custom domain is included in the SAN list. Then, you can use rpk on your local machine to communicate with the cluster externally using the self-signed certificate for encryption.

1. Configure external access to your Redpanda cluster using a custom domain.

   note

   Your Redpanda brokers should advertise addresses in your custom domain.

2. Install rpk on your local machine, not on a Pod:

   Linux

   1. Download the rpk archive for Linux:

      curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip

   2. Ensure that you have the folder ~/.local/bin:

      mkdir -p ~/.local/bin

   3. Add it to your $PATH:

      export PATH="~/.local/bin:$PATH"

   4. Unzip the rpk files to your ~/.local/bin/ directory:

      unzip rpk-linux-amd64.zip -d ~/.local/bin/

   5. Run rpk version to display the rpk binary version:

      rpk version
      Example output

      v22.3.11 (rev 9eefb907c)

   macOS

   1. If you don't have Homebrew installed, install it.

   2. Install rpk:

      brew install redpanda-data/tap/redpanda

   3. Run rpk version to display the rpk binary version:

      rpk version
      Example output

      v22.3.11 (rev 9eefb907c)

3. Save the root certificate authority (CA) to your local file system outside Kubernetes:

   kubectl -n redpanda get secret redpanda-default-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt

4. Pass the root CA to rpk to validate your external connection to Redpanda.

   Replace the following placeholders:

• <subdomain>: The subdomain that's in the advertised address of one of your Redpanda brokers.

• <custom-domain>: Your domain.

• <external-port>: The port on which your cluster is exposed.

  rpk cluster info \
    --brokers <subdomain>.<custom-domain>:<external-kafka-api-port> \
    --tls-truststore ca.crt

Disable TLS

If you disable TLS, Redpanda communicates over a plain-text network connection, where any malicious party can see all communication.

To disable TLS, set tls.enabled to false:

--values

no-tls.yaml

tls:
  enabled: false

helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --values self-signed-tls.yaml --reuse-values

--set

helm upgrade --install redpanda redpanda/redpanda -n redpanda --create-namespace \
  --set tls.enabled=false

Troubleshooting

invalid large response size

This error appears when you don't specify that you are connecting over TLS. For example:

kubectl exec redpanda-0 -c redpanda -n redpanda -- rpk cluster info \
  --brokers <subdomain>.<domain>:<external-kafka-api-port>

Result:

unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v1.2; is this a plaintext connection speaking to a tls endpoint?

Solution

Make sure to add the --tls-enabled option:

kubectl exec redpanda-0 -c redpanda -n redpanda -- rpk cluster info --brokers <subdomain>.<domain>:<external-kafka-api-port> --tls-enabled

i/o timeout

This error appears when your worker nodes are unreachable through the given address.

Solution

Check the following:

• The address and port are correct.

• Your DNS records point to addresses that resolve to your worker nodes.

Next steps

Add client authentication by combining TLS encryption with SASL authentication.