Redpanda Security on Kubernetes
You can configure TLS inside a Kubernetes cluster to establish a secure connection with encrypted communication between a client and a broker. If you need flexibility for authorization combined with encrypted communication, you can configure TLS encryption along with SASL authentication. Or, you can configure mTLS to authenticate both the server and the client.
For details about Kubernetes configuration, see:
For general information about Redpanda security, see:
Certificates
The Redpanda operator uses cert-manager to generate certificates for TLS and mTLS authentication. (SASL does not use certificates.) When the client opens a connection to Redpanda, Redpanda sends the client a certificate and the client verifies the certificate with the Certificate Authority. If mTLS is enabled, the client then sends its own certificate to Redpanda and Redpanda verifies that certificate with the Certificate Authority.
The Redpanda operator uses the following certificates:
Certificate | Description |
---|---|
Root |
Issuer certificate |
Node |
Redpanda node credentials |
Client |
Credentials used by the client |
If you delete the certificate, the secret does not get deleted. This means that if you delete the certificate manually, the operator continues to use the same secret. For information about re-creating the secret, see Certificate Resources. |
Root certificate
When you configure TLS or mTLS on a Redpanda cluster and you do not provide an issuer, the Redpanda operator uses cert-manager to generate a root certificate that is local to the cluster. The operator then uses the root certificate to generate a node certificate for the listener, and for mTLS a certificate is also created for the client. If you do provide an issuer, the operator does not generate a root certificate.
Node certificate
The operator provides the node certificate to Redpanda. The certificate secret is mounted as a volume that is consumed by Redpanda. For information about mounting secrets as a volume, see Secrets.
The node certificate secret is named in the following way for each API:
API | Secret name |
---|---|
Kafka API |
|
HTTP Proxy |
|
Schema Registry |
|
Admin API |
|
Client certificate (mTLS only)
The client certificate is generated when mTLS authentication is enabled. The client certificate is held by the client so that the server can use it to verify that the client is safe.
The client certificate secret is named in the following way for each API:
API | Secret name |
---|---|
Kafka API |
|
HTTP Proxy |
|
Schema Registry |
|
Admin API |
|
Provide a trusted certificate issuer or certificate
For Kafka API and Schema Registry, you have the option of providing a trusted certificate issuer or a certificate. For example, Redpanda Cloud uses a Let’s Encrypt issuer, which prevents the need for the client to to download the certificate for the cluster. Instead, the Let’s Encrypt certificate, which is available on all operating systems, is used by the client.
To provide an issuer, add the issuerRef
property to the Redpanda configuration file:
issuerRef:
name: <issuer_name>
kind: <issuer>
The issuerRef
property contains the following variables:
Variable | Description |
---|---|
|
Name of the issuer or cluster issuer. |
|
Kubernetes resource that represents a certificate authority. The value of this property can be |
To provide a certificate, add the nodeSecretRef
property to the Redpanda configuration file:
nodeSecretRef:
name: <secret_name>
namespace: <secret_namespace>
The nodeSecretRef
property contains the following variables:
Variable | Description |
---|---|
|
Name of the certificate secret. |
|
Kubernetes namespace where the certificate secret is. If the secret is in a different namespace than the Redpanda cluster, the operator copies it to the namespace of the Redpanda cluster. |
For details about providing a trusted issuer or certificate issuer, see Configuring Redpanda TLS on Kubernetes and Configuring Redpanda mTLS on Kubernetes.
Certificate secrets
The Redpanda operator uses cert-manager to generate certificates. When a certificate is created, a Kubernetes secret resource for the certificate is also created in the same namespace as the Redpanda cluster. The secret resource is the following type:
kubernetes.io/tls
For information about the kubernetes.io/tls
secret type, see TLS Secrets.
The kubernetes.io/tls
resource contains the following components:
-
tls.key
-
tls.crt
-
ca.crt
- This is provided if you’re using a self-signed Certificate Authority (that is, you didn’t provide an issuer in the cluster configuration file).
These components are described in TLS certificates with external connectivity and mTLS certificates with external connectivity.
To see the contents of kubernetes.io/tls
, run:
kubectl get secret <secret_name> -o yaml
Renew certificates
The certificate renewal process is handled seamlessly by cert-manager. You don’t need to do anything to facilitate the renewal. However, keep in mind that if you have a customer using the certificate, you need to give the new certificate to them. For that reason, a new certificate is issued 30 days before the old certificate expires. In this 30-day window, the new certificate and the old certificate are active, which gives you time to update the certificate.
The Redpanda operator sets the certificate duration to five years. This is non-configurable.
To see when your certificate was issued, when a new certificate will be issued, and when your certificate will expire, run:
kubectl describe certificate <certificate_name>
For information about renewing your certificate, see cert-manager documentation.
Subject Alternative Name
Each certificate has a Subject Alternative Name (SAN) that lists the DNS names secured by the certificate. When the Redpanda operator provides the certificate to the client, it provides the SAN. The SAN is structured like this:
DNS: *.<cluster_name>.default.svc.cluster.local
The wildcard (*) prefix indicates that the SAN is for all brokers. Redpanda does not generate certificates that are specific to brokers.
The client must specify a broker when it communicates with the operator. For example, the client might use this SAN:
DNS: 0.<cluster_name>.default.svc.cluster.local
For external connectivity, the SAN is structured like this:
DNS: *.<subdomain_name>
External connectivity
If the client is within the same Kubernetes cluster as Redpanda, you don’t need to configure external connectivity. However, if you have communication from outside the cluster or from outside the virtual private cloud, you need to set up external connectivity. This section contains an overview of how external connectivity works.
Listeners
The listener ports are the ports that the Redpanda APIs use to communicate with the client. You must configure external connectivity on each API individually. The following table lists the supported listener configurations for each API with TLS enabled.
API | Listener configurations with TLS |
---|---|
Kafka API |
<ul><li> One internal listener with TLS enabled </li><li> One internal listener and one external listener. Only one of the listeners can have TLS enabled. </li> </ul> |
Schema Registry |
<ul><li> One internal listener with TLS enabled </li><li> One listener that is used for internal and external connectivity with TLS enabled </li> </ul> |
You can specify up to two listeners for each API, but only one listener can have TLS or mTLS enabled. If you do have two listeners, one must be external. The exception is Schema Registry, which can only have one listener. The Schema Registry listener can be internal, or it can be an internal port that is used internally and externally. If you enable external connectivity on Schema Registry, the Kubernetes node port connects to the internal Redpanda port to provide external connectivity.
When you configure external connectivity, you can specify the external port, but you don’t need to. If you do not specify a port, a port is picked from the 3000-32767
range. This range is the default specified in Kubernetes.
For information about the autogenerated port and directions on how to change the default range, see Type NodePort.
Configure external connectivity
To enable external connectivity with TLS, add the following lines to each API in the configuration file:
- external:
enabled: true
subdomain: <subdomain_name>
The external port is generated automatically and you don’t need to specify it. In the following example, TLS is enabled on the external listener for the Kafka API. Enable external connectivity the same way for the Admin API and HTTP Proxy.
kafkaApi:
- port: 9092
- external:
enabled: true
subdomain: <subdomain_name>
tls:
enabled: true
The Schema Registry syntax is slightly different, in that the ports are not a list. You can specify one internal port and one external port. Schema Registry always uses an internal port and with external connectivity configured, the Kubernetes node port connects to the internal Redpanda port. Configure TLS with external connectivity for Schema Registry like this:
schemaRegistry:
port: 8081
external:
enabled: true
subdomain: <subdomain_name>
tls:
enabled: true
Subdomain
The subdomain
field lets you specify the advertised address of the external listener. The subdomain addresses, including the brokers, must be registered with a DNS provider, such as Amazon Route 53. Each API in the configuration file must have the same subdomain
specified. The configuration file uses the subdomain
field to generate the advertised addresses for the external listeners.
The advertised addresses for the external listeners are structured like this:
<broker_id>.\<subdomain_name>:\<node_port>
If you don’t provide a subdomain, you can’t configure TLS or mTLS for the cluster. The Redpanda operator does not issue certificates for IP addresses.
TLS certificates with external connectivity
If you have external connectivity configured for your cluster and you didn’t provide an issuer in the configuration file, you must export the Certificate Authority’s (CA) public certificate file from the node certificate Secret as a file named ca.crt
.
To extract ca.crt
from the certificate Secret, run:
kubectl get secret <secret_name> -o go-template='{{index .data "ca.crt"}}' | base64 -d - > ca.crt
The secret names for each API are listed in Node certificate. |
After you have ca.crt
extracted, you can create an rpk
configuration file that lets you to run rpk
from anywhere, including outside the Kubernetes cluster. To do this, create a configuration file with the following content:
rpk:
kafka_api:
brokers:
- 0.<subdomain_name>.:<node_port>
- 1.<subdomain_name>.:<node_port>
- 2.<subdomain_name>.:<node_port>
tls:
truststore_file: <truststore_file_path>/ca.crt
The file contains the following variables:
Variable | Description |
---|---|
|
The |
|
Port on the machine where Redpanda is listening. If you don’t specify this port in the cluster configuration file, the port is autogenerated. |
|
Directory where you want to mount the |
You can reference the configuration file in your rpk
commands when you manage the Redpanda cluster. The --config
flag lets you specify a Redpanda configuration file.
For example, if you saved the file as pandas_config.yaml
, you can create a topic called panda
with this command:
rpk topic create panda --config pandas_config.yaml
mTLS certificates with external connectivity
If you have external connectivity configured for your cluster and you’re using mTLS, you must extract the tls.crt
and tls.key
files from the client certificate secret and export them to the client. In addition, if you didn’t provide an issuer in the cluster configuration file, you must export ca.crt
.
The following table gives the command to extract each of these files.
The secret names for each API are listed in Node certificate. |
File | Description | Command to extract the file |
---|---|---|
|
CA certificate (only created if you do not specify an issuer) |
|
|
Client certificate |
|
|
Private client key |
|
The following command retrieves the entire resource to view the contents. But keep in mind that the Kafka client cannot process the resource as a single file.
|
After you have ca.crt
, tls.crt
, and tls.key
extracted, you can create an rpk
configuration file that lets you run rpk
from anywhere, including outside the Kubernetes cluster. To do this, create a configuration file with the following content:
rpk:
kafka_api:
brokers:
- 0.<subdomain_name>.:<node_port>
- 1.<subdomain_name>.:<node_port>
- 2.<subdomain_name>.:<node_port>
tls:
key_file: <key_file_path>/tls.key
cert_file: <cert_file_path>/tls.crt
truststore_file: <truststore_file_path>/ca.crt
The file contains the following variables:
Variable | Description |
---|---|
|
The |
|
Port on the machine where Redpanda is listening. If you don’t specify this port in the cluster configuration file, the port is autogenerated. |
|
Directory where you want to mount the |
|
Filename and directory where you want to mount the |
|
Directory where you want to mount the |
You can reference the configuration file in your rpk
commands when you manage the Redpanda cluster. The --config
flag lets you specify a Redpanda configuration file.
For example, if you saved the file as pandas_config.yaml
, you can create a topic called pandas
in the cluster with this command:
rpk topic create triceratops --config triceratops_config.yaml