Streaming

Rotate TLS certificates in Kubernetes

Rotate the TLS certificates that Redpanda uses in Kubernetes before they expire, so that brokers, the Redpanda Operator, and clients keep connecting securely without unplanned downtime.

After reading this page, you will be able to:

Configure how long cert-manager certificates for Redpanda remain valid
Rotate the root CA of the Helm chart’s self-signed issuer without breaking client connections
Replace certificates stored in your own Kubernetes Secrets

Prerequisites

A Redpanda cluster in Kubernetes with TLS enabled. See Use cert-manager to manage TLS certificates or Use Kubernetes Secrets to manage TLS certificates.
kubectl, configured to communicate with your cluster.
For certificates managed by cert-manager, the cmctl command-line tool. You use it to trigger certificate renewal on demand.
The openssl command-line tool, to inspect certificates.

The examples on this page use the default certificate names for a Helm release named redpanda. If your release has a different name, replace the redpanda- prefix in Certificate and Secret names. Replace <namespace> with the namespace of your Redpanda cluster.

How Redpanda loads renewed certificates

The Helm chart mounts each certificate’s Secret into the Redpanda container as a volume at /etc/tls/certs/<cert-name>. When the contents of a Secret change, the kubelet updates the mounted files. This update is not immediate: it can take up to the kubelet sync period plus the kubelet’s cache propagation delay. See the Kubernetes documentation.

Redpanda watches its certificate, key, and truststore files and reloads them when they change, without a restart. This applies to the Kafka API, Admin API, HTTP Proxy, Schema Registry, and internal RPC listeners. When a reload succeeds, the broker logs a message that contains key or certificate file updated.

The chart’s StatefulSet does not restart Pods when the contents of a certificate Secret change. A change to the StatefulSet’s Pod template, such as pointing a certificate at a different Secret, triggers a rolling restart.

To confirm that brokers loaded the new files, search the broker logs:

kubectl logs redpanda-0 --namespace <namespace> -c redpanda | grep "key or certificate file updated"

You can also monitor the redpanda_tls_loaded_at_timestamp_seconds metric, and alert on redpanda_tls_certificate_expires_at_timestamp_seconds and redpanda_tls_truststore_expires_at_timestamp_seconds to catch certificates and CAs that are close to expiring. See TLS metrics.

Renew certificates managed by cert-manager

When the Helm chart uses cert-manager, cert-manager renews each Certificate resource automatically and updates its Secret. You don’t need to take action for renewed broker certificates: Redpanda reloads them as described in How Redpanda loads renewed certificates.

For each certificate in tls.certs, the duration setting controls how long issued certificates remain valid. The default is 43800h (five years). The chart doesn’t set renewBefore on its Certificate resources, so cert-manager uses its default and renews each certificate two-thirds of the way through its duration. The chart doesn’t expose a setting for renewBefore.

To change the validity period, set duration on the certificate:

Operator
Helm

redpanda-cluster.yaml

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    tls:
      certs:
        default:
          duration: 8760h

kubectl apply -f redpanda-cluster.yaml --namespace <namespace>

--values
--set

cert-duration.yaml

tls:
  certs:
    default:
      duration: 8760h

helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --values cert-duration.yaml --reuse-values

helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set tls.certs.default.duration=8760h

The value must end in h, m, or s.

When the chart uses its self-signed issuer, the duration setting applies to both the root CA and the certificates that the CA signs. When you change duration, cert-manager reissues these Certificates right away, including the root CA. Renewing the root CA requires extra steps, so after you change duration, follow Rotate the CA of the self-signed issuer.

When you set issuerRef to use your own Issuer or ClusterIssuer, your issuer controls the CA. The chart creates only the broker and client Certificates, and cert-manager renews them automatically.

To check when cert-manager will next renew a certificate, run:

kubectl get certificate redpanda-default-cert --namespace <namespace> \
  -o go-template='{{ .status.notAfter }}{{ "\n" }}{{ .status.renewalTime }}{{ "\n" }}'

Rotate the CA of the self-signed issuer

By default, the Helm chart bootstraps a CA for each certificate in tls.certs that has no issuerRef or secretRef:

A self-signing Issuer, such as redpanda-default-selfsigned-issuer.
A root CA Certificate, such as redpanda-default-root-certificate, signed by the self-signing Issuer.
A CA Issuer, such as redpanda-default-root-issuer, that signs the broker certificate (redpanda-default-cert) and, with mTLS, the client certificate (redpanda-default-client) using the root CA.

cert-manager renews the root CA Certificate automatically, but renewing a CA does not reissue the certificates that the CA signed. Each broker certificate’s Secret keeps the ca.crt of the CA that signed it. If the old CA expires before the broker certificate is reissued, clients and brokers that verify against that ca.crt fail TLS verification, and the Redpanda Pods can fail their readiness probes. When the broker certificate renews on its own schedule after the root CA renews, the new CA signs it, and clients that trust only the previous CA fail TLS verification.

If you deploy with the Redpanda Operator, the operator verifies the brokers against the new root CA as soon as it renews. Until the brokers load a certificate that the new CA signed, the operator logs TLS verification errors (x509: certificate signed by unknown authority) and cannot finish reconciling the cluster. Complete the following steps without delay after the root CA renews.

To avoid this, reissue the broker and client certificates after the root CA renews, and make sure that clients trust the new CA. Use the following steps after cert-manager renews the root CA automatically, or to rotate the CA on your own schedule.

Check the expiration dates of the root CA and of the CA in the broker certificate’s Secret:

kubectl get secret redpanda-default-root-certificate --namespace <namespace> \
  -o go-template='{{ index .data "tls.crt" | base64decode }}' | openssl x509 -noout -fingerprint -sha256 -enddate

kubectl get secret redpanda-default-cert --namespace <namespace> \
  -o go-template='{{ index .data "ca.crt" | base64decode }}' | openssl x509 -noout -fingerprint -sha256 -enddate

If the fingerprints differ, cert-manager already renewed the root CA, and the broker certificate still references the previous CA. Skip step 3.

Save a copy of the CA that signed the current broker certificate. Clients need it during the transition.

kubectl get secret redpanda-default-cert --namespace <namespace> \
  -o go-template='{{ index .data "ca.crt" | base64decode }}' > old-ca.crt

Renew the root CA:

cmctl renew redpanda-default-root-certificate --namespace <namespace>

Wait until the Certificate is ready:

kubectl get certificate redpanda-default-root-certificate --namespace <namespace>

Give clients a CA bundle that contains both the previous CA and the new CA, so that they trust certificates from either CA during the transition:
```
kubectl get secret redpanda-default-root-certificate --namespace <namespace> \
  -o go-template='{{ index .data "tls.crt" | base64decode }}' > new-ca.crt
cat old-ca.crt new-ca.crt > ca-bundle.crt
```
Update each client’s trust store with ca-bundle.crt. For example, pass the bundle to rpk with -X tls.ca=ca-bundle.crt. Whether clients that trust only the previous CA continue to work after the next step depends on whether cert-manager generated a new private key for the CA. cert-manager 1.18 and later generate a new private key on each renewal by default. See the cert-manager rotationPolicy documentation.
Reissue the certificates that the root CA signs:
```
cmctl renew redpanda-default-cert --namespace <namespace>
```
If mTLS is enabled for listeners that use this certificate, also reissue the client certificate:
```
cmctl renew redpanda-default-client --namespace <namespace>
```
Verify that the ca.crt in the broker certificate’s Secret matches the new root CA. Repeat the commands in step 1 and compare the fingerprints.
Verify that the brokers reloaded the new files. See How Redpanda loads renewed certificates.
If Redpanda Console is deployed with your cluster, restart it so that its Kafka API and Schema Registry clients trust the new CA:
```
kubectl rollout restart deployment/redpanda-console --namespace <namespace>
```
Until it restarts, Redpanda Console fails TLS verification when it opens new connections to the brokers.
Repeat these steps for each certificate in tls.certs that uses the self-signed issuer, such as external.
After all clients use the new CA, you can remove the previous CA from their trust stores.

Rotate certificates in your own Secrets

When you provide certificates in a Secret with tls.certs.<cert-name>.secretRef, you are responsible for renewing them. To rotate a certificate, update the contents of the existing Secret. Redpanda reloads the new files without a restart, as described in How Redpanda loads renewed certificates.

Create the new certificate and key. Include the same subject alternative names (SANs) as the certificate you are replacing.

Replace the contents of the existing Secret.

If your certificates are issued by a public CA:

kubectl create secret tls <secret-name> \
  --cert=<path>tls.crt \
  --key=<path>tls.key \
  --namespace <namespace> \
  --dry-run=client -o yaml | kubectl apply -f -

If your certificates are issued by a private CA:

kubectl create secret generic <secret-name> \
  --from-file=<path>tls.crt \
  --from-file=<path>tls.key \
  --from-file=<path>ca.crt \
  --namespace <namespace> \
  --dry-run=client -o yaml | kubectl apply -f -

Replace the <path> placeholders with the paths to your certificate files.

Verify that the brokers reloaded the new files. See How Redpanda loads renewed certificates.

If you rotate a private CA, Redpanda uses the ca.crt file in the Secret as the truststore for listeners that require client authentication when caEnabled is true. To keep accepting clients during the transition, put both the previous CA and the new CA in ca.crt, then remove the previous CA after all client certificates are reissued.

If you create a Secret with a different name instead of updating the existing one, update tls.certs.<cert-name>.secretRef.name to the new Secret. This changes the StatefulSet’s Pod template, which triggers a rolling restart of the brokers.

Next steps

Suggested labs

Search all labs

Was this helpful?

group Ask in the community

mail Share your feedback