Use cert-manager to manage TLS certificates

When using cert-manager for TLS certificate management, you can use a self-signed certificate or a certificate signed by a trusted certificate authority (CA). This topic provides instructions for each option.

Redpanda supports both TLS and mTLS:

  • TLS, previously SSL, provides encryption for client-server communication. A server certificate prevents third parties from accessing data transferred between the client and the server.

  • mTLS, or mutual TLS, is a protocol that authenticates both the server and the client. In addition to the server certificate required in TLS, mTLS also requires the client to give a certificate. mTLS is useful for environments that require additional security and only have a small number of verified clients.

Prerequisites

You must have the following:

  • Kubernetes cluster: Ensure you have a running Kubernetes cluster, either locally, such as with minikube or kind, or remotely.

  • Kubectl. Ensure you have the kubectl command-line tool installed and configured to communicate with your cluster.

  • cert-manager. Ensure you have cert-manager and its custom resource definitions (CRDs) installed.

  • If you want to connect to your Redpanda cluster from outside Kubernetes, make sure to enable external access.

Use a self-signed certificate

By default, the Redpanda Helm chart uses cert-manager to generate four Certificate resources. These resources provide Redpanda brokers with self-signed TLS certificates for internal and external listeners.

Type Default Certificates

Internal

  • redpanda-default-cert: Self-signed certificate for internal communications.

  • redpanda-default-root-certificate: Root CA for the internal certificate.

External

  • redpanda-external-cert: Self-signed certificate for external communications.

  • redpanda-external-root-certificate: Root CA for the external certificate.

A corresponding Secret resource exists for each Certificate resource. The Secret contains the TLS files.

Having separate self-signed certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications.

The following steps explain how to set up self-signed certificates in your Redpanda cluster:

  1. Make sure that TLS is enabled:

    • Helm + Operator

    • Helm

    redpanda-cluster.yaml
    apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
    kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

    • --values

    • --set

    self-signed-tls.yaml
    tls:
  enabled: true
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values self-signed-tls.yaml --reuse-values
    helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true

    Enabling or disabling TLS for the RPC listener requires you to delete all Pods that run Redpanda. When you change the rpc.tls.enabled setting, or if it is not overridden and you change the global tls.enabled option, Redpanda cannot safely apply the change because RPC listener configurations must be the same across all brokers. To apply the change, all Redpanda Pods must be deleted simultaneously so that they all start with the updated RPC listener. This action results in temporary downtime of the cluster.

    Although you can use the --force option to speed up the rollout, it may result in data loss as Redpanda will not be given time to shut down gracefully.

    kubectl delete pod -l app=redpanda --namespace <namespace>

  2. Make sure the Certificates are in a READY state.

    kubectl get certificate --namespace <namespace>
    NAME                                 READY
redpanda-default-cert                True
redpanda-default-root-certificate    True
redpanda-external-cert               True
redpanda-external-root-certificate   True

Connect to Redpanda

You can use the rpk command-line client to test both internal and external connections to Redpanda.

Test internal connections

Your self-signed certificate’s SAN includes the internal addresses assigned to brokers by Kubernetes through the headless ClusterIP Service. As such, you can use rpk within the Redpanda container to securely communicate with the cluster.

You can validate your internal connection to Redpanda with rpk by executing the following command:

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info
Expected output: 
CLUSTER
=======
redpanda.19ae8532-c8fa-49ed-8b35-82d74813db3a

BROKERS
=======
ID    HOST                                                PORT
0*    redpanda-0.redpanda.<namespace>.svc.cluster.local.  9093
1     redpanda-.redpanda.<namespace>.svc.cluster.local.   9093
2     redpanda-2.redpanda.<namespace>.svc.cluster.local.  9093

Test external connections

To test external connections, you must enable external access using a custom domain. See Prerequisites.

The SAN list of your self-signed certificate does not contain the IP addresses of your worker nodes, but when you enable external access using a custom domain, that domain is included in the SAN list. Then, you can use rpk on your local machine to communicate with the cluster externally using the self-signed certificate for encryption.

  1. Configure external access to your Redpanda cluster using a custom domain.

    Your Redpanda brokers should advertise addresses in your custom domain.

  2. Install rpk on your local machine, not inside the container:

    • Linux

    • macOS

    • amd64

    • arm64

    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip &&
  mkdir -p ~/.local/bin &&
  export PATH="~/.local/bin:$PATH" &&
  unzip rpk-linux-amd64.zip -d ~/.local/bin/
    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip &&
  mkdir -p ~/.local/bin &&
  export PATH="~/.local/bin:$PATH" &&
  unzip rpk-linux-arm64.zip -d ~/.local/bin/

    1. If you don’t have Homebrew installed, install it.

    2. To install or update rpk, run:

      brew install redpanda-data/tap/redpanda
      This method installs the latest version of rpk, which is supported only with the latest version of Redpanda.

  3. Save the external root CA to your local file system outside Kubernetes:

    kubectl --namespace <namespace> get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt

  4. Configure rpk to connect to your cluster using the pre-configured profile:

    rpk profile create --from-profile <(kubectl get configmap --namespace <namespace> redpanda-rpk -o go-template='{{ .data.profile }}') <profile-name>

  5. Test the connection:

    rpk cluster info

Use a public CA certificate

Certificates from a public CA are trusted by default. You can configure the Helm chart to use an Issuer resource or a ClusterIssuer resource to generate publicly trusted Certificates for external connections. These custom resources are managed by cert-manager.

The Issuer or ClusterIssuer specifies the CA that will be used when generating certificates. If you select an ACME server such as Let’s Encrypt as the CA, cert-manager automatically handles the required HTTP01 or DNS01 ACME challenges to issue certificates.

  1. Create an Issuer in the same namespace as your Redpanda cluster, or create a ClusterIssuer in any namespace. For details, see the cert-manager documentation.

  2. Configure the Helm chart with your Issuer or ClusterIssuer.

    Replace the following placeholders:

    • <issuer-name>: The name of your Issuer or ClusterIssuer resource.

    • <issuer>: Issuer or ClusterIssuer.

    • <custom-domain>: Your domain.

      • Helm + Operator

      • Helm

      redpanda-cluster.yaml
      apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        external:
          issuerRef:
            name: <issuer-name>
            kind: <issuer>
          caEnabled: false
    external:
      domain: <custom-domain>
      kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

      • --values

      • --set

      external-tls.yaml
      tls:
  enabled: true
  certs:
    external:
      issuerRef:
        name: <issuer-name>
        kind: <issuer>
      caEnabled: false
external:
  domain: <custom-domain>
      helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values external-tls.yaml
      helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.external.issuerRef.name=<issuer-name> \
  --set tls.certs.external.issuerRef.kind=<issuer> \
  --set tls.certs.external.caEnabled=false \
  --set external.domain=<custom-domain>

      Enabling or disabling TLS for the RPC listener requires you to delete all Pods that run Redpanda. When you change the rpc.tls.enabled setting, or if it is not overridden and you change the global tls.enabled option, Redpanda cannot safely apply the change because RPC listener configurations must be the same across all brokers. To apply the change, all Redpanda Pods must be deleted simultaneously so that they all start with the updated RPC listener. This action results in temporary downtime of the cluster.

      Although you can use the --force option to speed up the rollout, it may result in data loss as Redpanda will not be given time to shut down gracefully.

      kubectl delete pod -l app=redpanda --namespace <namespace>

  3. Make sure the Certificates are in a READY state.

    kubectl get certificate --namespace <namespace>

By default, this certificate is used to encrypt traffic between clients and all external listeners. You can select specific certificates for each external listener. See Configure Listeners in Kubernetes.

Connect to Redpanda

You can use the rpk command-line client to test both internal and external connections to Redpanda.

Test internal connections

Validate your internal connection to Redpanda with rpk by executing the following command.

kubectl exec redpanda-0 --namespace <namespace> -c redpanda -- rpk cluster info

You should see the Kafka API endpoints for the internal listener. For example:

CLUSTER
=======
redpanda.271dac90-2dc8-48e4-9dc6-652f63684d73

BROKERS
=======
ID    HOST                                             PORT
0*    redpanda-0.redpanda.redpanda.svc.cluster.local.  9093
1     redpanda-1.redpanda.redpanda.svc.cluster.local.  9093
2     redpanda-2.redpanda.redpanda.svc.cluster.local.  9093

Kubernetes assigns the *.redpanda.redpanda.svc.cluster.local. DNS names to the brokers through the headless ClusterIP Service. These are internal Kubernetes addresses. Port 9093 is the default port of the internal listener for the Kafka API.

Test external connections

To test external connections, external access must be enabled on your cluster and your brokers must advertise an address that’s resolvable externally by your clients.

To test external connections:

  1. Install rpk on your local machine, not inside the container:

    • Linux

    • macOS

    • amd64

    • arm64

    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip &&
  mkdir -p ~/.local/bin &&
  export PATH="~/.local/bin:$PATH" &&
  unzip rpk-linux-amd64.zip -d ~/.local/bin/
    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip &&
  mkdir -p ~/.local/bin &&
  export PATH="~/.local/bin:$PATH" &&
  unzip rpk-linux-arm64.zip -d ~/.local/bin/

    1. If you don’t have Homebrew installed, install it.

    2. To install or update rpk, run:

      brew install redpanda-data/tap/redpanda
      This method installs the latest version of rpk, which is supported only with the latest version of Redpanda.

  2. If your TLS certificates were issued by a private CA, save the root CA to your local file system outside Kubernetes:

    kubectl --namespace <namespace> get secret <secret-name> -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt

  3. Configure rpk to connect to your cluster using the pre-configured profile:

    rpk profile create --from-profile <(kubectl get configmap --namespace <namespace> redpanda-rpk -o go-template='{{ .data.profile }}') <profile-name>

  4. Test the connection:

    rpk cluster info

You should see the Kafka API endpoints for the external listener. For example:

CLUSTER
=======
redpanda.271dac90-2dc8-48e4-9dc6-652f63684d73

BROKERS
=======
ID    HOST                                   PORT
0*    redpanda-0.customredpandadomain.local  31092
1     redpanda-1.customredpandadomain.local  31092
2     redpanda-2.customredpandadomain.local  31092

The Helm chart configures brokers with these addresses using the values of the external.domain and/or external.addresses settings in the Helm values. These are external addresses that external clients use to connect. Port 31092 is the default node port of the external listener for the Kafka API. This node port is assigned to the Kubernetes Services.

If your brokers external addresses are not resolvable, you can test external connections by sending API requests to the container port that’s assigned to the external listener. For example:

kubectl exec redpanda-0 -n redpanda -c redpanda -- rpk cluster info -X brokers=redpanda-0.redpanda.redpanda.svc.cluster.local:9094 -X tls.ca=/etc/tls/certs/external/ca.crt

Port 9094 is the default container port for the external Kafka API.

For more details on connecting to Redpanda, see Connect to Redpanda in Kubernetes.

Manage the minimum TLS version

Redpanda sets the minimum TLS version for all clusters to 1.2, using the tls_min_version cluster configuration property. This prevents client applications from negotiating a downgrade to the TLS version when they make a connection to a Redpanda cluster.

You can update the minimum TLS version of your clusters to v1.0, v1.1 or v1.3 using rpk. Replace the placeholder in brackets.

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    config:
      cluster:
        tls_min_version: <version-number>
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

  • --values

  • --set

tls-version.yaml
config:
  cluster:
    tls_min_version: <version-number>
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values tls-version.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set config.cluster.tls_min_version=<version-number>

Configure mTLS encryption

Mutual TLS (mTLS), is an enhanced security protocol that extends the standard TLS encryption protocol. While standard TLS involves a server presenting a certificate to the client to prove its identity, mTLS adds an additional layer of security by requiring the client to also present a certificate to the server.

To enable mTLS, set the requireClientAuth setting to true for all listeners.

listeners:
  kafka:
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
  admin:
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
  schemaRegistry:
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
  rpc:
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
  http:
    tls:
      enabled: true
      cert: default
      requireClientAuth: true

When you enable mTLS, the Helm chart generates a TLS Secret resource called redpanda-client that contains the client’s tls.crt and tls.key files. If you want to use your own TLS files, you can instead provide the Helm chart with a custom TLS Secret or Issuer (if you’re using cert-manager) to use in tls.certs and then reference its name in your listener configuration.

All listeners must use the same certificate if mTLS is enabled. Redpanda on Kubernetes does not support configuring different TLS certificates for mTLS-enabled listeners, or mixed configurations where some listeners use mTLS and others do not. 
listeners:
  kafka:
    tls:
      enabled: true
      cert: <custom-cert-secret>
      requireClientAuth: true
  admin:
    tls:
      enabled: true
      cert: <custom-cert-secret>
      requireClientAuth: true
  schemaRegistry:
    tls:
      enabled: true
      cert: <custom-cert-secret>
      requireClientAuth: true
  rpc:
    tls:
      enabled: true
      cert: <custom-cert-secret>
      requireClientAuth: true
  http:
    tls:
      enabled: true
      cert: <custom-cert-secret>
      requireClientAuth: true

Configure a truststore

To ensure that mTLS functions correctly, it is important to properly configure the truststore.

The truststore contains the CA certificates that the server uses to validate the client certificates. By explicitly setting the truststore, you ensure that mTLS will use the correct CA certificates to validate client certificates, enhancing the security and reliability of your Redpanda cluster.

To set a truststore for a listener, you can use a Secret or a ConfigMap reference:

  • Helm + Operator

  • Helm

redpanda-cluster.yaml
apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      enabled: true
      certs:
        default:
          caEnabled: true
    listeners:
      admin:
        tls:
          trustStore:
            SecretKeyRef:
              key: <key-name>
              name: <secret-name>
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

  • --values

  • --set

redpanda-tls.yaml
tls:
  enabled: true
  certs:
    default:
      caEnabled: true
listeners:
  admin:
    tls:
      trustStore:
        SecretKeyRef:
          key: <key-name>
          name: <secret-name>
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values redpanda-tls.yaml --reuse-values
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.enabled=true \
  --set tls.certs.default.caEnabled=true \
  --set listeners.admin.tls.trustStore.SecretKeyRef.key=<key-name> \
  --set listeners.admin.tls.trustStore.SecretKeyRef.name=<secret-name>

Disable TLS

If you disable TLS, Redpanda communicates over a plain-text network connection, where any malicious party can see all communication.

To disable TLS for all listeners, set tls.enabled to false:

no-tls.yaml
tls:
  enabled: false

To disable TLS for a specific listener, set tls.enabled to false for the listener. For example, to disable TLS for the internal Kafka API listener:

listeners:
  kafka:
    tls:
      enabled: false

Enabling or disabling TLS for the RPC listener requires you to delete all Pods that run Redpanda. When you change the rpc.tls.enabled setting, or if it is not overridden and you change the global tls.enabled option, Redpanda cannot safely apply the change because RPC listener configurations must be the same across all brokers. To apply the change, all Redpanda Pods must be deleted simultaneously so that they all start with the updated RPC listener. This action results in temporary downtime of the cluster.

Although you can use the --force option to speed up the rollout, it may result in data loss as Redpanda will not be given time to shut down gracefully.

kubectl delete pod -l app=redpanda --namespace <namespace>

Troubleshoot

Here are some common troubleshooting scenarios and their solutions. For more troubleshooting steps, see Troubleshoot Redpanda in Kubernetes.

I/O timeout

This error appears when your worker nodes are unreachable through the given address.

Check the following:

  • The address and port are correct.

  • Your DNS records point to addresses that resolve to your worker nodes.

Redpanda not applying TLS changes

Enabling or disabling TLS for the RPC listener requires you to delete all Pods that run Redpanda. When you change the rpc.tls.enabled setting, or if it is not overridden and you change the global tls.enabled option, Redpanda cannot safely apply the change because RPC listener configurations must be the same across all brokers. To apply the change, all Redpanda Pods must be deleted simultaneously so that they all start with the updated RPC listener. This action results in temporary downtime of the cluster.

Although you can use the --force option to speed up the rollout, it may result in data loss as Redpanda will not be given time to shut down gracefully.

kubectl delete pod -l app=redpanda --namespace <namespace>

Invalid large response size

This error appears when your cluster is configured to use TLS, but you don’t specify that you are connecting over TLS.

unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v1.2; is this a plaintext connection speaking to a tls endpoint?

If you’re using rpk, ensure to add the -X tls.enabled flag, and any other necessary TLS flags such as the TLS certificate:

kubectl exec <pod-name> -c redpanda --namespace <namespace> -- \
rpk cluster info -X tls.enabled=true

For all available flags, see the rpk options reference.

Malformed HTTP response

This error appears when a cluster has TLS enabled, and you try to access the admin API without passing the required TLS parameters.

Retrying POST for error: Post "http://127.0.0.1:9644/v1/security/users": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x03\x00\x02\x02"

If you’re using rpk, ensure to include the TLS flags.

For all available flags, see the rpk options reference.

x509: certificate signed by unknown authority

This error appears when the Certificate Authority (CA) that signed your certificates is not trusted by your system.

Check the following:

  • Ensure you have installed the root CA certificate correctly on your local system.

  • If using a self-signed certificate, ensure it is properly configured and included in your system’s trust store.

  • If you are using a certificate issued by a CA, ensure the issuing CA is included in your system’s trust store.

  • If you are using cert-manager, ensure it is correctly configured and running properly.

  • Check the validity of your certificates. They might have expired.

x509: certificate is not valid for any names

This error indicates that the certificate you are using is not valid for the specific domain or IP address you are trying to use it with. This error typically occurs when there is a mismatch between the certificate’s Subject Alternative Name (SAN) or Common Name (CN) field and the name being used to access the broker.

To fix this error, you may need to obtain a new certificate that is valid for the specific domain or IP address you are using. Ensure that the certificate’s SAN or CN entry matches the name being used, and that the certificate is not expired or revoked.

cannot validate certificate for 127.0.0.1

This error appears if you are using a CA certificate when you try to establish an internal connection using localhost. For example:

unable to request metadata: unable to dial: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

To fix this error, you must either specify the URL with a public domain or use self-signed certificates:

kubectl exec redpanda-0 -c redpanda --namespace <namespace> -- \
rpk cluster info \
-X brokers=<redpanda-url>:<port> \
-X tls.enabled=true

Next steps

Suggested reading