# Use cert-manager to manage TLS certificates > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Use cert-manager to manage TLS certificates latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "true" page-eol-date: December 22, 2024 latest-console-tag: v3.7.3 latest-connect-version: 4.93.0 docname: kubernetes/security/tls/k-cert-manager page-component-name: streaming page-version: "23.3" page-component-version: "23.3" page-component-title: Streaming page-relative-src-path: kubernetes/security/tls/k-cert-manager.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/v/23.3/modules/manage/pages/kubernetes/security/tls/k-cert-manager.adoc description: Learn how to enable TLS encryption in your Redpanda cluster and use cert-manager to simplify the process of obtaining, renewing, and using certificates. page-git-created-date: "2024-01-04" page-git-modified-date: "2024-02-26" support-status: past end-of-life --- When using [cert-manager](https://cert-manager.io/docs/) for TLS certificate management, you have the option to use a self-signed certificate or a certificate signed by a trusted certificate authority (CA). This topic provides instructions for each option. TLS, previously SSL, provides encryption for client-server communication. A server certificate prevents third parties from accessing data transferred between the client and server. ## [](#prerequisites)Prerequisites You must have the following: - Kubernetes cluster: Ensure you have a running Kubernetes cluster, either locally, such as with minikube or kind, or remotely. - [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). Ensure you have the `kubectl` command-line tool installed and configured to communicate with your cluster. - [cert-manager](https://cert-manager.io/docs/installation/helm/). Ensure you have cert-manager and its custom resource definitions (CRDs) installed. - If you want to connect to your Redpanda cluster from outside Kubernetes, make sure to [enable external access](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/external/). ## [](#use-a-self-signed-certificate)Use a self-signed certificate By default, the Redpanda Helm chart uses cert-manager to generate four Certificate resources. These resources provide Redpanda brokers with self-signed TLS certificates for internal and external listeners. | Type | Default Certificates | | --- | --- | | Internal | redpanda-default-cert: Self-signed certificate for internal communications.redpanda-default-root-certificate: Root CA for the internal certificate. | | External | redpanda-external-cert: Self-signed certificate for external communications.redpanda-external-root-certificate: Root CA for the external certificate. | For each Certificate resource, a corresponding Secret resource exists, which contains the TLS files. Having separate self-signed certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications. The following steps explain how to set up self-signed certificates in your Redpanda cluster: 1. Make sure that TLS is enabled: ### Helm + Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: tls: enabled: true ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` ### Helm #### --values `self-signed-tls.yaml` ```yaml tls: enabled: true ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values self-signed-tls.yaml --reuse-values ``` #### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set tls.enabled=true ``` 2. Make sure the Certificates are in a `READY` state. ```bash kubectl get certificate --namespace ``` NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True ## [](#connect-to-redpanda)Connect to Redpanda You can use the `rpk` command-line client to test both internal and external connections to Redpanda. ### [](#test-internal-connections)Test internal connections Your self-signed certificate’s SAN includes the internal addresses assigned to brokers by Kubernetes through the headless ClusterIP Service. As such, you can use `rpk` within the Redpanda container to securely communicate with the cluster. You can validate your internal connection to Redpanda with `rpk` by executing the following command: ```bash kubectl exec redpanda-0 --namespace -c redpanda -- rpk cluster info ``` Expected output: ```none CLUSTER ======= redpanda.19ae8532-c8fa-49ed-8b35-82d74813db3a BROKERS ======= ID HOST PORT 0* redpanda-0.redpanda..svc.cluster.local. 9093 1 redpanda-.redpanda..svc.cluster.local. 9093 2 redpanda-2.redpanda..svc.cluster.local. 9093 ``` ### [](#test-external-connections)Test external connections To test external connections, you must enable external access using a custom domain. See [Prerequisites](#prerequisites). The SAN list of your self-signed certificate does not contain the IP addresses of your worker nodes, but when you enable external access using a custom domain, that domain is included in the SAN list. Then, you can use `rpk` on your local machine to communicate with the cluster externally using the self-signed certificate for encryption. 1. Configure [external access](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/external/) to your Redpanda cluster using a custom domain. > 📝 **NOTE** > > Your Redpanda brokers should advertise addresses in your custom domain. 2. Install `rpk` on your local machine, not inside the container: #### Linux 1. Download the `rpk` archive for Linux, and make sure the version matches your Redpanda version. - To download the latest version of `rpk`: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip ``` - To download a version other than the latest: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-amd64.zip ``` 2. Ensure that you have the folder `~/.local/bin`: ```bash mkdir -p ~/.local/bin ``` 3. Add it to your `$PATH`: ```bash export PATH="~/.local/bin:$PATH" ``` 4. Unzip the `rpk` files to your `~/.local/bin/` directory: ```bash unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` 5. Run `rpk version` to display the version of the rpk binary: ```bash rpk version ``` ```bash 23.3.1 (rev b5ade3f40) ``` #### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. Install `rpk`: ```bash brew install redpanda-data/tap/redpanda ``` 3. Run `rpk version` to display the version of the rpk binary: ```bash rpk version ``` ```bash 23.3.1 (rev b5ade3f40) ``` > 📝 **NOTE** > > This method installs the latest version of `rpk`, which is supported only with the latest version of Redpanda. 3. Save the external root CA to your local file system outside Kubernetes: ```bash kubectl --namespace get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt ``` 4. Configure `rpk` to connect to your cluster using the [pre-configured profile](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` 5. Test the connection: ```bash rpk cluster info ``` ## [](#use-a-public-ca-certificate)Use a public CA certificate Certificates from a public CA are trusted by default. You can configure the Helm chart to use an Issuer resource or a ClusterIssuer resource to generate publicly trusted Certificates for external connections. These custom resources are managed by cert-manager. The Issuer or ClusterIssuer specifies the CA that will be used when generating certificates. If you select an ACME server such as Let’s Encrypt as the CA, cert-manager automatically handles the required HTTP01 or DNS01 ACME challenges to issue certificates. 1. Create an Issuer in the same namespace as your Redpanda cluster, or create a ClusterIssuer in any namespace. For details, see the [cert-manager documentation](https://cert-manager.io/docs/configuration/). 2. Configure the Helm chart with your Issuer or ClusterIssuer. Replace the following placeholders: - ``: The name of your Issuer or ClusterIssuer resource. - ``: `Issuer` or `ClusterIssuer`. - ``: Your domain. ### Helm + Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: tls: enabled: true certs: external: issuerRef: name: kind: caEnabled: false external: domain: ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` ### Helm #### --values `external-tls.yaml` ```yaml tls: enabled: true certs: default: issuerRef: name: kind: caEnabled: false external: domain: ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values external-tls.yaml ``` #### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set tls.enabled=true \ --set tls.certs.default.issuerRef.name= \ --set tls.certs.default.issuerRef.kind= \ --set tls.certs.default.caEnabled=false \ --set external.domain= ``` By default, this certificate will be used to encrypt traffic between clients and all external listeners. You can select specific certificates for each external listener. See [Configure Listeners in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/k-configure-listeners/#tls). 3. Make sure the Certificates are in a `READY` state. ```bash kubectl get certificate --namespace ``` ## [](#connect-to-redpanda-2)Connect to Redpanda You can use the `rpk` command-line client to test both internal and external connections to Redpanda. ### [](#test-internal-connections-2)Test internal connections Validate your internal connection to Redpanda with `rpk` by executing the following command. ```bash kubectl exec redpanda-0 --namespace -c redpanda -- rpk cluster info ``` You should see the Kafka API endpoints for the internal listener. For example: CLUSTER ======= redpanda.271dac90-2dc8-48e4-9dc6-652f63684d73 BROKERS ======= ID HOST PORT 0\* redpanda-0.redpanda.redpanda.svc.cluster.local. 9093 1 redpanda-1.redpanda.redpanda.svc.cluster.local. 9093 2 redpanda-2.redpanda.redpanda.svc.cluster.local. 9093 Kubernetes assigns the `*.redpanda.redpanda.svc.cluster.local.` DNS names to the brokers through the headless ClusterIP Service. These are internal Kubernetes addresses. Port 9093 is the default port of the internal listener for the Kafka API. ### [](#test-external-connections-2)Test external connections To test external connections, external access must be enabled on your cluster and your brokers must advertise an address that’s resolvable externally by your clients. To test external connections: 1. Install `rpk` on your local machine, not inside the container: #### Linux 1. Download the `rpk` archive for Linux, and make sure the version matches your Redpanda version. - To download the latest version of `rpk`: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip ``` - To download a version other than the latest: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-amd64.zip ``` 2. Ensure that you have the folder `~/.local/bin`: ```bash mkdir -p ~/.local/bin ``` 3. Add it to your `$PATH`: ```bash export PATH="~/.local/bin:$PATH" ``` 4. Unzip the `rpk` files to your `~/.local/bin/` directory: ```bash unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` 5. Run `rpk version` to display the version of the rpk binary: ```bash rpk version ``` ```bash 23.3.1 (rev b5ade3f40) ``` #### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. Install `rpk`: ```bash brew install redpanda-data/tap/redpanda ``` 3. Run `rpk version` to display the version of the rpk binary: ```bash rpk version ``` ```bash 23.3.1 (rev b5ade3f40) ``` > 📝 **NOTE** > > This method installs the latest version of `rpk`, which is supported only with the latest version of Redpanda. 2. If your TLS certificates were issued by a private CA, save the root CA to your local file system outside Kubernetes: ```bash kubectl --namespace get secret -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt ``` 3. Configure `rpk` to connect to your cluster using the [pre-configured profile](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` 4. Test the connection: ```bash rpk cluster info ``` You should see the Kafka API endpoints for the external listener. For example: CLUSTER ======= redpanda.271dac90-2dc8-48e4-9dc6-652f63684d73 BROKERS ======= ID HOST PORT 0\* redpanda-0.customredpandadomain.local 31092 1 redpanda-1.customredpandadomain.local 31092 2 redpanda-2.customredpandadomain.local 31092 The Helm chart configures brokers with these addresses using the values of the `external.domain` and/or `external.addresses` settings in the Helm values. These are external addresses that external clients use to connect. Port 31092 is the default node port of the external listener for the Kafka API. This node port is assigned to the Kubernetes Services. If your brokers external addresses are not resolvable, you can test external connections by sending API requests to the container port that’s assigned to the external listener. For example: ```bash kubectl exec redpanda-0 -n redpanda -c redpanda -- rpk cluster info -X brokers=redpanda-0.redpanda.redpanda.svc.cluster.local:9094 -X tls.ca=/etc/tls/certs/external/ca.crt ``` Port 9094 is the default container port for the external Kafka API. For more details on connecting to Redpanda, see [Connect to Redpanda in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/k-connect-to-redpanda/). ## [](#disable-tls)Disable TLS If you disable TLS, Redpanda communicates over a plain-text network connection, where any malicious party can see all communication. To disable TLS for all listeners, set `tls.enabled` to `false`: `no-tls.yaml` ```yaml tls: enabled: false ``` To disable TLS for a specific listener, set `tls.enabled` to `false` for the listener. For example, to disable TLS for the internal Kafka API listener: ```yaml listeners: kafka: tls: enabled: false ``` ## [](#troubleshoot)Troubleshoot Here are some common troubleshooting scenarios and their solutions. For more troubleshooting steps, see [Troubleshoot Redpanda in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/troubleshooting/k-troubleshoot/). ### [](#invalid-large-response-size)Invalid large response size This error appears when your cluster is configured to use TLS, but you don’t specify that you are connecting over TLS. unable to request metadata: invalid large response size 352518912 > limit 104857600; the first three bytes received appear to be a tls alert record for TLS v1.2; is this a plaintext connection speaking to a tls endpoint? If you’re using rpk, ensure to add the `-X tls.enabled` flag, and any other necessary TLS flags such as the TLS certificate: ```bash kubectl exec -c redpanda --namespace -- rpk cluster info -X brokers=.: -X tls.enabled=true ``` For all available flags, see the [rpk command reference](https://docs.redpanda.com/streaming/23.3/reference/rpk/). ### [](#malformed-http-response)Malformed HTTP response This error appears when a cluster has TLS enabled, and you try to access the admin API without passing the required TLS parameters. Retrying POST for error: Post "http://127.0.0.1:9644/v1/security/users": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\\x15\\x03\\x03\\x00\\x02\\x02" If you’re using rpk, ensure to include the TLS flags. For all available flags, see the [rpk command reference](https://docs.redpanda.com/streaming/23.3/reference/rpk/). ### [](#x509-certificate-signed-by-unknown-authority)x509: certificate signed by unknown authority This error appears when the Certificate Authority (CA) that signed your certificates is not trusted by your system. Check the following: - Ensure you have installed the root CA certificate correctly on your local system. - If using a self-signed certificate, ensure it is properly configured and included in your system’s trust store. - If you are using a certificate issued by a CA, ensure the issuing CA is included in your system’s trust store. - If you are using cert-manager, ensure it is correctly configured and running properly. - Check the validity of your certificates. They might have expired. ### [](#x509-certificate-is-not-valid-for-any-names)x509: certificate is not valid for any names This error indicates that the certificate you are using is not valid for the specific domain or IP address you are trying to use it with. This error typically occurs when there is a mismatch between the certificate’s Subject Alternative Name (SAN) or Common Name (CN) field and the name being used to access the broker. To fix this error, you may need to obtain a new certificate that is valid for the specific domain or IP address you are using. Ensure that the certificate’s SAN or CN entry matches the name being used, and that the certificate is not expired or revoked. ### [](#cannot-validate-certificate-for-127-0-0-1)cannot validate certificate for 127.0.0.1 This error appears if you are using a CA certificate when you try to establish an internal connection using localhost. For example: ```none unable to request metadata: unable to dial: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs ``` To fix this error, you must either specify the public domain or use self-signed certificates: ```bash kubectl exec redpanda-0 -c redpanda --namespace -- \ rpk cluster info \ -X brokers=.: \ -X tls.enabled=true ``` ### [](#io-timeout)I/O timeout This error appears when your worker nodes are unreachable through the given address. Check the following: - The address and port are correct. - Your DNS records point to addresses that resolve to your worker nodes. ## [](#next-steps)Next steps - [Configure Authentication for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/security/authentication/k-authentication/) - [Configure Listeners in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/k-configure-listeners/) - [Connect to Redpanda in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/networking/k-connect-to-redpanda/) ## [](#suggested-reading)Suggested reading - [Securing Redpanda in Kubernetes (Day 2 Ops)](https://killercoda.com/redpanda/scenario/redpanda-k8s-secure) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/23.3/reference/k-redpanda-helm-spec/#external) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/23.3/reference/k-crd/) ## Suggested labs - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) [Search all labs](https://docs.redpanda.com/labs)