23.2 23.1 22.3 22.2 22.1 21.11

Configure TLS for Redpanda in Kubernetes

Redpanda clusters use Transport Layer Security (TLS) to secure internal and external communication. TLS is enabled globally by default for all listeners, using self-signed certificates managed by cert-manager. This topic provides guidance on configuring TLS with or without cert-manager, covering the use of self-signed certificates as well as Certificate Authority (CA) certificates.

Use cert-manager to manage TLS certificates

When using cert-manager for TLS certificate management, you have the option to use a self-signed certificate or a CA certificate. The following sections provide detailed instructions for each option.

Ensure you have cert-manager installed.

Use a self-signed certificate

A self-signed certificate is signed with its own private key, instead of a public or private certificate authority (CA). The following steps provide the required configurations and commands for setting up self-signed certificates in your Redpanda cluster:

  1. Make sure that TLS is enabled (default):

    • Helm + Operator

    • Helm

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

    • --values

    • --set

    self-signed-tls.yaml
    tls:
  enabled: true
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values self-signed-tls.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true

  2. Make sure the Certificates are in a READY state.

    kubectl get certificate --namespace <namespace>
    NAME                                 READY   SECRET                               AGE
redpanda-default-cert                True    redpanda-default-cert                10m
redpanda-default-root-certificate    True    redpanda-default-root-certificate    10m

Test internal connections

Your self-signed certificate’s SAN list includes the internal addresses of the ClusterIP Service. As such, you can use rpk within the Redpanda container to securely communicate with the cluster internally using the self-signed certificate for encryption.

You can validate your internal connection to Redpanda with rpk by executing the following command:

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info \
  --tls-enabled \
  -X brokers=<broker-url>:<kafka-api-port>\
  --tls-truststore <path-to-ca-certificate>
Expected output: 
CLUSTER
=======
redpanda.19ae8532-c8fa-49ed-8b35-82d74813db3a

BROKERS
=======
ID    HOST                                             PORT
0*    redpanda-0.redpanda.<namespace>.svc.cluster.local.  9093
1     redpanda-.redpanda.<namespace>.svc.cluster.local.  9093
2     redpanda-2.redpanda.<namespace>.svc.cluster.local.  9093

Test external connections

To test external connections, you must enable external access using a custom domain. The SAN list of your self-signed certificate does not contain the IP addresses of your worker nodes, but when you enable external access using a custom domain, that domain is included in the SAN list. Then, you can use rpk on your local machine to communicate with the cluster externally using the self-signed certificate for encryption.

  1. Configure external access to your Redpanda cluster using a custom domain.

    Your Redpanda brokers should advertise addresses in your custom domain.

  2. Install rpk on your local machine, not on a Pod:

    • Linux

    • macOS

    1. Download the rpk archive for Linux, and make sure the version matches your Redpanda version.

      • To download the latest version of rpk:

        curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip

      • To download a version other than the latest:

        curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-linux-amd64.zip

    2. Ensure that you have the folder ~/.local/bin:

      mkdir -p ~/.local/bin

    3. Add it to your $PATH:

      export PATH="~/.local/bin:$PATH"

    4. Unzip the rpk files to your ~/.local/bin/ directory:

      unzip rpk-linux-amd64.zip -d ~/.local/bin/

    5. Run rpk version to display the version of the rpk binary:

      rpk version
      23.2.9 (rev 045d510)

    1. If you don’t have Homebrew installed, install it.

    2. Install rpk:

      brew install redpanda-data/tap/redpanda

    3. Run rpk version to display the version of the rpk binary:

      rpk version
      23.2.9 (rev 045d510)
      This method installs the latest version of rpk, which is supported only with the latest version of Redpanda.

  3. Save the root certificate authority (CA) to your local file system outside Kubernetes:

    kubectl --namespace <namespace> get secret redpanda-default-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt

  4. Pass the root CA to rpk to validate your external connection to Redpanda.

    Replace the following placeholders:

    • <subdomain>: The subdomain that’s in the advertised address of one of your Redpanda brokers.

    • <custom-domain>: Your domain.

    • <external-port>: The port on which your cluster is exposed.

      rpk cluster info \
  -X brokers=<subdomain>.<custom-domain>:<external-kafka-api-port> \
  --tls-truststore ca.crt

Use a public CA certificate

Certificates from a public certificate authority (CA) are trusted by default. You can configure the Helm chart to use an Issuer or ClusterIssuer custom resource to generate publicly trusted Certificates. These custom resources are managed by cert-manager.

The Issuer or ClusterIssuer specifies the CA that will be used when generating certificates. If an ACME server such as Let’s Encrypt is chosen as the CA, cert-manager will automatically handle the required HTTP01 or DNS01 ACME challenges to issue certificates.

  1. Create an Issuer in the same namespace as your Redpanda cluster, or create a ClusterIssuer in any namespace. For details, see the cert-manager documentation.

  2. Configure the Helm chart with your Issuer or ClusterIssuer.

    Replace the following placeholders:

    • <issuer-name>: The name of your Issuer or ClusterIssuer resource.

    • <issuer>: Issuer or ClusterIssuer.

    • <custom-domain>: Your domain.

      • Helm + Operator

      • Helm

      redpanda-cluster.yaml
      apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        default:
          issuerRef:
            name: <issuer-name>
            kind: <issuer>
          caEnabled: false
    external:
      domain: <custom-domain>
      kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

      • --values

      • --set

      ca-tls.yaml
      tls:
  enabled: true
  certs:
    default:
      issuerRef:
        name: <issuer-name>
        kind: <issuer>
      caEnabled: false
external:
  domain: <custom-domain>
      helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values ca-tls.yaml
      helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.default.issuerRef.name=<issuer-name> \
  --set tls.certs.default.issuerRef.kind=<issuer> \
  --set tls.certs.default.caEnabled=false \
  --set external.domain=<custom-domain>

  3. Make sure the Certificates are in a READY state.

    kubectl get certificate --namespace <namespace>
NAME                                 READY   SECRET                               AGE
redpanda-default-cert                True    redpanda-default-cert                10m
redpanda-default-root-certificate    True    redpanda-default-root-certificate    10m

Test internal connections

The SAN list of your CA certificate includes only your custom domains and subdomains. You can use rpk inside the redpanda container to communicate with the cluster internally using your domain as the address.

Validate your internal connection to Redpanda with rpk:

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info -X brokers=<subdomain>.<custom-domain>:<kafka-api-port>
Expected output: 
CLUSTER
=======
redpanda.19ae8532-c8fa-49ed-8b35-82d74813db3a

BROKERS
=======
ID    HOST                                             PORT
0*    redpanda-0.redpanda.<namespace>.svc.cluster.local.  9093
1     redpanda-.redpanda.<namespace>.svc.cluster.local.  9093
2     redpanda-2.redpanda.<namespace>.svc.cluster.local.  9093

Test external connections

Validate your external connection to Redpanda with rpk:

Replace the following placeholders:

  • <subdomain>: The subdomain that’s in the advertised address of one of your Redpanda brokers.

  • <custom-domain>: Your domain.

  • <external-port>: The port on which your cluster’s Kafka API is exposed.

rpk cluster info -X brokers=<subdomain>.<custom-domain>:<external-port>

Use Secrets to manage TLS certificates without cert-manager

If you prefer not to use cert-manager for TLS certificate management, you can manage your TLS configurations using Kubernetes Secrets. The following steps outline how to configure TLS with Secrets:

  1. Create a Kubernetes Secret that contains your TLS certificates.

    • To create a Secret that includes your tls.crt and tls.key files:

      kubectl create secret tls <secret-name> \
  --cert=<path>tls.crt \
  --key=<path>tls.key \
  --namespace <namespace>

    • To include the ca.crt file, you must create an Opaque Secret:

      kubectl create secret generic <secret-name> \
  --from-file=<path>tls.crt \
  --from-file=<path>tls.key \
  --from-file=<path>ca.crt \
  --namespace <namespace>
      When using certificates issued by public certificate authorities (CAs), you don’t need to provide the ca.crt file in the Secret. Public CAs are already trusted by default in most systems and web browsers. The trust chain is built into the operating system or web browser, which includes the root certificates of well-known CAs.

      Replace the <path> placeholders with the paths to your certificate files.

      For an example of creating the TLS certificates, see the GitHub repository.

  2. Update your Redpanda Helm configuration to use the Secret:

    • Helm + Operator

    • Helm

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        default:
          secretRef:
            name: <secret-name>

    If you are using a private CA, set caEnabled to true.

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        default:
          secretRef:
            name: <secret-name>
          caEnabled: true
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

    • --values

    • --set

    tls-secret.yml
    tls:
  enabled: true
  certs:
    default:
      secretRef:
        name: <secret-name>

    If you are using a private CA, set caEnabled to true.

    tls-secret.yml
    tls:
  enabled: true
  certs:
    default:
      secretRef:
        name: <secret-name>
      caEnabled: true
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values tls-secret.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.default.secretRef.name=<secret-name>

    If you are using a private CA, set caEnabled to true.

    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.default.secretRef.name=<secret-name> \
  --set tls.certs.default.caEnabled=true

Test internal connections

You can validate your internal connection to Redpanda with rpk by executing the following command.

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info \
  --tls-enabled \
  -X brokers=<broker-url>:<kafka-api-port>

If you are using a private CA, include the path to your CA certificate in the --tls-truststore option.

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info \
  --tls-enabled \
  -X brokers=<broker-url>:<kafka-api-port> \
  --tls-truststore <path-to-ca-certificate>

Test external connections

To test external connections, you need to enable external access using a custom domain. You can use rpk on your local machine to communicate with the cluster externally using the TLS certificates for encryption.

rpk cluster info \
  -X brokers=<subdomain>.<custom-domain>:<external-kafka-api-port>

If you are using a private CA, include the path to your CA certificate in the --tls-truststore option.

rpk cluster info \
  -X brokers=<subdomain>.<custom-domain>:<external-kafka-api-port> \
  --tls-truststore <path-to-ca-certificate>

Use multiple certificates for different listeners

The Redpanda cluster provides granular control over the TLS certificates used by different listeners. This level of flexibility enables you to ensure the required level of security for each listener. For example, you can use a self-signed certificate for the internal RPC listener, while using a Certificate Authority (CA) certificate for other listeners such as the Kafka API.

By default, all listeners in Redpanda use the self-signed certificate defined globally in the tls.certs.default configuration. To customize the certificates for each listener, you can edit the listeners.<listener-name>.tls.cert setting.

If you don’t want to generate the self-signed certificate, set tls.certs.default to null.

Here’s an example that configures two certificates: public-ca-cert and private-ca-cert. The public-ca-cert certificate is configured with an Issuer managed by cert-manager, while the private-ca-cert certificate is configured with a generic Secret containing the tls.crt, tls.key, and ca.crt files.

The Admin API listener is configured with the public-ca-cert certificate, the Kafka API listener is configured with the private-ca-cert certificate, and the other listeners are configured with the default self-signed certificate.

multiple-certs-tls.yaml
tls:
  enabled: true
  certs:
    public-ca-cert:
      issuerRef:
        name: <issuer-name>
        kind: Issuer
      caEnabled: false
    private-ca-cert:
      secretRef:
        name: <secret-name>
      caEnabled: true
    default:
      caEnabled: true
listeners:
  admin:
    tls:
      cert: public-ca-cert
  kafka:
    tls:
      cert: private-ca-cert
  http:
    tls:
      cert: default
  rpc:
    tls:
      cert: default
  schemaRegistry:
    tls:
      cert: default

Disable TLS

If you disable TLS, Redpanda communicates over a plain-text network connection, where any malicious party can see all communication.

To disable TLS, set tls.enabled to false:

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: false
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
--values
no-tls.yaml
tls:
  enabled: false
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values self-signed-tls.yaml --reuse-values
--set
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=false

Troubleshooting

Here are some common troubleshooting scenarios and their solutions:

invalid large response size

This error appears when you don’t specify that you are connecting over TLS. For example:

kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rpk cluster info \
  -X brokers=<subdomain>.<domain>:<external-kafka-api-port>

Result:

unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v.2; is this a plaintext connection speaking to a tls endpoint?

Solution:

Make sure to add the --tls-enabled option:

kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rpk cluster info -X brokers=<subdomain>.<domain>:<external-kafka-api-port> --tls-enabled

i/o timeout

This error appears when your worker nodes are unreachable through the given address.

Solution:

Check the following:

  • The address and port are correct.

  • Your DNS records point to addresses that resolve to your worker nodes.

cannot validate certificate for 127.0.0.1

This error appears if you are using a CA certificate when you try to establish an internal connection using localhost. For example:

kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rpk cluster info --tls-enabled

Result:

unable to request metadata: unable to dial: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

Solution:

You must specify the public domain:

kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- rpk cluster info -X brokers=<subdomain>.<domain>:<external-port> --tls-enabled

Or, use self-signed certificates.

x509: certificate signed by unknown authority

This error appears when the Certificate Authority (CA) that signed your certificates is not trusted by your system.

Solution:

Check the following:

  • Make sure you have installed the Root CA certificate correctly on your local system.

  • If using a self-signed certificate, ensure it is properly configured and included in your system’s trust store.

  • If you are using a certificate issued by a CA, make sure the issuing CA is included in your system’s trust store.

  • If you are using cert-manager, ensure it is correctly configured and running properly.

  • Check the validity of your certificates. They might have expired.

x509: certificate is not valid for any names

This error indicates that the certificate you are using is not valid for the specific domain or IP address you are trying to use it with. This error typically occurs when there is a mismatch between the certificate’s Subject Alternative Name (SAN) or Common Name (CN) field and the name being used to access the broker.

To resolve this issue, you may need to obtain a new certificate that is valid for the specific domain or IP address you are using. Ensure that the certificate’s SAN or CN entry matches the name being used, and make sure the certificate is not expired or revoked.

Next steps

Add client authentication by combining TLS encryption with SASL authentication.

Suggested reading