23.3 Beta 23.2 23.1 22.3 22.2 22.1 21.11

Redpanda Security on Kubernetes

You can configure TLS inside a Kubernetes cluster to establish a secure connection with encrypted communication between a client and a broker. If you need flexibility for authorization combined with encrypted communication, you can configure TLS encryption along with SASL authentication. Or, you can configure mTLS to authenticate both the server and the client.

For details about Kubernetes configuration, see:

For general information about Redpanda security, see:

Certificates

The Redpanda operator uses cert-manager to generate certificates for TLS and mTLS authentication. (SASL does not use certificates.) When the client opens a connection to Redpanda, Redpanda sends the client a certificate and the client verifies the certificate with the Certificate Authority. If mTLS is enabled, the client then sends its own certificate to Redpanda and Redpanda verifies that certificate with the Certificate Authority.

The Redpanda operator uses the following certificates:

Certificate Description

Root

Issuer certificate

Node

Redpanda node credentials

Client

Credentials used by the client
If you delete the certificate, the secret does not get deleted. This means that if you delete the certificate manually, the operator continues to use the same secret. For information about re-creating the secret, see Certificate Resources.

Root certificate

When you configure TLS or mTLS on a Redpanda cluster and you do not provide an issuer, the Redpanda operator uses cert-manager to generate a root certificate that is local to the cluster. The operator then uses the root certificate to generate a node certificate for the listener, and for mTLS a certificate is also created for the client. If you do provide an issuer, the operator does not generate a root certificate.

Node certificate

The operator provides the node certificate to Redpanda. The certificate secret is mounted as a volume that is consumed by Redpanda. For information about mounting secrets as a volume, see Secrets.

The node certificate secret is named in the following way for each API:

API Secret name

Kafka API

-redpanda</code>

HTTP Proxy

-proxy-api-node</code>

Schema Registry

-schema-registry-node</code>

Admin API

-admin-api-node</code>

Client certificate (mTLS only)

The client certificate is generated when mTLS authentication is enabled. The client certificate is held by the client so that the server can use it to verify that the client is safe.

The client certificate secret is named in the following way for each API:

API Secret name

Kafka API

-user-client</code>

HTTP Proxy

-proxy-api-client</code>

Schema Registry

-schema-registry-client</code>

Admin API

-admin-api-client</code>

Provide a trusted certificate issuer or certificate

For Kafka API and Schema Registry, you have the option of providing a trusted certificate issuer or a certificate. For example, Redpanda Cloud uses a Let’s Encrypt issuer, which prevents the need for the client to to download the certificate for the cluster. Instead, the Let’s Encrypt certificate, which is available on all operating systems, is used by the client.

To provide an issuer, add the issuerRef property to the Redpanda configuration file:

issuerRef:
  name: <issuer_name>
  kind: <issuer>

The issuerRef property contains the following variables:

Variable Description

issuer_name

Name of the issuer or cluster issuer.

issuer

Kubernetes resource that represents a certificate authority. The value of this property can be Issuer or ClusterIssuer. If the kind property is not set, or if it is set to Issuer, an issuer with the name specified in the name property that exists in the same namespace as the certificate is used.

To provide a certificate, add the nodeSecretRef property to the Redpanda configuration file:

nodeSecretRef:
  name: <secret_name>
  namespace: <secret_namespace>

The nodeSecretRef property contains the following variables:

Variable Description

secret_name

Name of the certificate secret.

secret_namespace

Kubernetes namespace where the certificate secret is. If the secret is in a different namespace than the Redpanda cluster, the operator copies it to the namespace of the Redpanda cluster.

For details about providing a trusted issuer or certificate issuer, see Configuring Redpanda TLS on Kubernetes and Configuring Redpanda mTLS on Kubernetes.

Certificate secrets

The Redpanda operator uses cert-manager to generate certificates. When a certificate is created, a Kubernetes secret resource for the certificate is also created in the same namespace as the Redpanda cluster. The secret resource is the following type:

kubernetes.io/tls

For information about the kubernetes.io/tls secret type, see TLS Secrets.

The kubernetes.io/tls resource contains the following components:

  • tls.key

  • tls.crt

  • ca.crt - This is provided if you’re using a self-signed Certificate Authority (that is, you didn’t provide an issuer in the cluster configuration file).

These components are described in TLS certificates with external connectivity and mTLS certificates with external connectivity.

To see the contents of kubernetes.io/tls, run:

kubectl get secret <secret_name> -o yaml

Renew certificates

The certificate renewal process is handled seamlessly by cert-manager. You don’t need to do anything to facilitate the renewal. However, keep in mind that if you have a customer using the certificate, you need to give the new certificate to them. For that reason, a new certificate is issued 30 days before the old certificate expires. In this 30-day window, the new certificate and the old certificate are active, which gives you time to update the certificate.

The Redpanda operator sets the certificate duration to five years. This is non-configurable.

To see when your certificate was issued, when a new certificate will be issued, and when your certificate will expire, run:

kubectl describe certificate <certificate_name>

For information about renewing your certificate, see cert-manager documentation.

Subject Alternative Name

Each certificate has a Subject Alternative Name (SAN) that lists the DNS names secured by the certificate. When the Redpanda operator provides the certificate to the client, it provides the SAN. The SAN is structured like this:

DNS: *.<cluster_name>.default.svc.cluster.local

The wildcard (*) prefix indicates that the SAN is for all brokers. Redpanda does not generate certificates that are specific to brokers.

The client must specify a broker when it communicates with the operator. For example, the client might use this SAN:

DNS: 0.<cluster_name>.default.svc.cluster.local

For external connectivity, the SAN is structured like this:

DNS: *.<subdomain_name>

External connectivity

If the client is within the same Kubernetes cluster as Redpanda, you don’t need to configure external connectivity. However, if you have communication from outside the cluster or from outside the virtual private cloud, you need to set up external connectivity. This section contains an overview of how external connectivity works.

Listeners

The listener ports are the ports that the Redpanda APIs use to communicate with the client. You must configure external connectivity on each API individually. The following table lists the supported listener configurations for each API with TLS enabled.

API Listener configurations with TLS

Kafka API
HTTP Proxy
Admin API

<ul><li> One internal listener with TLS enabled </li><li> One internal listener and one external listener. Only one of the listeners can have TLS enabled. </li> </ul>

Schema Registry

<ul><li> One internal listener with TLS enabled </li><li> One listener that is used for internal and external connectivity with TLS enabled </li> </ul>

You can specify up to two listeners for each API, but only one listener can have TLS or mTLS enabled. If you do have two listeners, one must be external. The exception is Schema Registry, which can only have one listener. The Schema Registry listener can be internal, or it can be an internal port that is used internally and externally. If you enable external connectivity on Schema Registry, the Kubernetes node port connects to the internal Redpanda port to provide external connectivity.

When you configure external connectivity, you can specify the external port, but you don’t need to. If you do not specify a port, a port is picked from the 3000-32767 range. This range is the default specified in Kubernetes.

For information about the autogenerated port and directions on how to change the default range, see Type NodePort.

Configure external connectivity

To enable external connectivity with TLS, add the following lines to each API in the configuration file:

  - external:
      enabled: true
      subdomain: <subdomain_name>

The external port is generated automatically and you don’t need to specify it. In the following example, TLS is enabled on the external listener for the Kafka API. Enable external connectivity the same way for the Admin API and HTTP Proxy.

kafkaApi:
  - port: 9092
  - external:
      enabled: true
      subdomain: <subdomain_name>
    tls:
      enabled: true

The Schema Registry syntax is slightly different, in that the ports are not a list. You can specify one internal port and one external port. Schema Registry always uses an internal port and with external connectivity configured, the Kubernetes node port connects to the internal Redpanda port. Configure TLS with external connectivity for Schema Registry like this:

schemaRegistry:
  port: 8081
  external:
    enabled: true
    subdomain: <subdomain_name>
  tls:
    enabled: true

Subdomain

The subdomain field lets you specify the advertised address of the external listener. The subdomain addresses, including the brokers, must be registered with a DNS provider, such as Amazon Route 53. Each API in the configuration file must have the same subdomain specified. The configuration file uses the subdomain field to generate the advertised addresses for the external listeners.

The advertised addresses for the external listeners are structured like this:

<broker_id>.\<subdomain_name>:\<node_port>

If you don’t provide a subdomain, you can’t configure TLS or mTLS for the cluster. The Redpanda operator does not issue certificates for IP addresses.

TLS certificates with external connectivity

If you have external connectivity configured for your cluster and you didn’t provide an issuer in the configuration file, you must export the Certificate Authority’s (CA) public certificate file from the node certificate Secret as a file named ca.crt.

To extract ca.crt from the certificate Secret, run:

kubectl get secret <secret_name> -o go-template='{{index .data "ca.crt"}}' | base64 -d - > ca.crt
The secret names for each API are listed in Node certificate.

After you have ca.crt extracted, you can create an rpk configuration file that lets you to run rpk from anywhere, including outside the Kubernetes cluster. To do this, create a configuration file with the following content:

rpk:
  kafka_api:
    brokers:
      - 0.<subdomain_name>.:<node_port>
      - 1.<subdomain_name>.:<node_port>
      - 2.<subdomain_name>.:<node_port>
    tls:
      truststore_file: <truststore_file_path>/ca.crt

The file contains the following variables:

Variable Description

subdomain_name

The subdomain_name that you specified in the cluster configuration file.

node_port

Port on the machine where Redpanda is listening. If you don’t specify this port in the cluster configuration file, the port is autogenerated.

truststore_file_path

Directory where you want to mount the ca.crt file. Generally this is /etc/tls/certs/ca.

You can reference the configuration file in your rpk commands when you manage the Redpanda cluster. The --config flag lets you specify a Redpanda configuration file.

For example, if you saved the file as pandas_config.yaml, you can create a topic called panda with this command:

rpk topic create panda --config pandas_config.yaml

mTLS certificates with external connectivity

If you have external connectivity configured for your cluster and you’re using mTLS, you must extract the tls.crt and tls.key files from the client certificate secret and export them to the client. In addition, if you didn’t provide an issuer in the cluster configuration file, you must export ca.crt.

The following table gives the command to extract each of these files.

The secret names for each API are listed in Node certificate.
File Description Command to extract the file

ca.crt

CA certificate (only created if you do not specify an issuer)

kubectl get secret -o go-template='{{index .data "ca.crt"}}' base64 -d - > ca.crt</code>

tls.crt

Client certificate

kubectl get secret -o go-template='{{index .data "tls.crt"}}' base64 -d - > tls.crt</code>

tls.key

Private client key

kubectl get secret -o go-template='{{index .data "tls.key"}}' base64 -d - > tls.key</code>

The following command retrieves the entire resource to view the contents. But keep in mind that the Kafka client cannot process the resource as a single file.

kubectl get secret <secret_name> --namespace=default -o yaml

After you have ca.crt, tls.crt, and tls.key extracted, you can create an rpk configuration file that lets you run rpk from anywhere, including outside the Kubernetes cluster. To do this, create a configuration file with the following content:

rpk:
  kafka_api:
    brokers:
      - 0.<subdomain_name>.:<node_port>
      - 1.<subdomain_name>.:<node_port>
      - 2.<subdomain_name>.:<node_port>
    tls:
      key_file: <key_file_path>/tls.key
      cert_file: <cert_file_path>/tls.crt
      truststore_file: <truststore_file_path>/ca.crt

The file contains the following variables:

Variable Description

subdomain_name

The subdomain_name that you specified in the cluster configuration file.

node_port

Port on the machine where Redpanda is listening. If you don’t specify this port in the cluster configuration file, the port is autogenerated.

key_file_path

Directory where you want to mount the tls.key private client key. Generally this is /etc/tls/certs.

cert_file_path

Filename and directory where you want to mount the tls.crt private key. Generally this is /etc/tls/certs.

truststore_file_path

Directory where you want to mount the ca.crt file. Generally this is /etc/tls/certs/ca.

You can reference the configuration file in your rpk commands when you manage the Redpanda cluster. The --config flag lets you specify a Redpanda configuration file.

For example, if you saved the file as pandas_config.yaml, you can create a topic called pandas in the cluster with this command:

rpk topic create triceratops --config triceratops_config.yaml