Configure Kafka TLS Encryption
By default, Redpanda data is sent unencrypted. A security best practice is to enable encryption with TLS or mTLS.
-
Transport Layer Security (TLS), previously SSL, provides encryption for client-server communication. A server certificate prevents third parties from accessing data transferred between the client and server.
-
mTLS, or 2-way TLS, is a protocol that authenticates both the server and the client. In addition to the server certificate required in TLS, mTLS also requires the client to give a certificate. This involves more overhead to implement, but it can be useful for environments that require additional security and only have a small number of verified clients.
For each Redpanda broker, specify the key file (broker.key
), the certificate file (broker.crt
), and the truststore file (ca.crt
). Each broker has its own broker.key
and broker.crt
, but all brokers can have the same ca.crt
.
Prerequisites
TLS certificates are necessary for encryption. You can use your own certificates, either self-signed or issued by a trusted certificate authority (CA).
Create a local CA for self-signed certificates
This step is required if you want to generate multiple certificates all signed by the same root; for example, you want to use mTLS but issue different certificates to multiple Redpanda brokers and clients. |
To generate a self-signed certificate in a single command:
openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout broker.key -out broker.crt -subj "/CN=redpanda" -addext "subjectAltName = DNS:localhost, IP: 127.0.0.1"
1. Create CA configuration file (ca.cnf)
Edit the distinguished_name
section with your own organization details. For default_md
, sha256
is the minimum message digest level. The subjectAltName
must be accurate for the broker’s certificate.
[ ca ]
default_ca = CA_default
[ CA_default ]
default_days = 365
database = index.txt
serial = serial.txt
default_md = sha256
copy_extensions = copy
unique_subject = no
policy = signing_policy
[ signing_policy ]
organizationName = supplied
commonName = optional
# Used to create the CA certificate.
[ req ]
prompt = no
distinguished_name = distinguished_name
x509_extensions = extensions
[ distinguished_name ]
organizationName = Redpanda
commonName = Redpanda CA
[ extensions ]
keyUsage = critical,digitalSignature,nonRepudiation,keyEncipherment,keyCertSign
basicConstraints = critical,CA:true,pathlen:1
# Common policy for nodes and users.
[ signing_policy ]
organizationName = supplied
commonName = optional
Create a CA key to self-sign certificates:
openssl genrsa -out ca.key 2048
chmod 400 ca.key
Create a public certificate for the CA:
openssl req -new -x509 -config ca.cnf -key ca.key -days 365 -batch -out ca.crt
where:
Inputs | Description |
---|---|
|
New request. |
|
Create an X.509 certificate, instead of a certificate signing request (CSR). |
|
Configuration file to use when generating certificates (created above). |
|
Private key of the CA (created above). |
|
Number of days signed certificates are valid. |
|
Batch mode, where certificates are certified automatically. |
Output | Description |
---|---|
|
The public key certificate of the CA. |
Create certificate signing requests
This step creates the certificate signing request for the CA to extend trust over the broker’s certificates.
1. Create broker configuration file (broker.cnf)
A subject alternative name (SAN) indicates all domain names and IP addresses secured by the certificate. Depending on the address the client uses to connect to Redpanda, you might need to create a CNF file for each broker to modify the alt_names
section with organizational details. For production usage, edit alt_names
with DNS resolutions and/or the IP addresses.
[ req ]
prompt = no
distinguished_name = distinguished_name
req_extensions = extensions
[ distinguished_name ]
organizationName = Redpanda
[ extensions ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = localhost
DNS.2 = redpanda
DNS.3 = console
DNS.4 = connect
DNS.5 = ec2-3-15-15-272.us-east-2.compute.amazonaws.com
IP.1 = 10.0.8.1
You could configure alternative names with a single version of broker.key
/broker.crt
,
as long as you update the certificate for all brokers in the cluster any time you
edit an entry. For example:
[ alt_names ]
DNS.1 = broker1.example.com
DNS.2 = broker2.example.com
DNS.3 = broker3.example.com
Additionally, you can configure alternative names using the public or private IP addresses of all your brokers. For example:
[ alt_names ]
IP.1 = 10.0.8.1
IP.2 = 10.0.8.2
IP.3 = 10.0.8.3
2. Generate broker private key (broker.key)
Generate a 2048-bit RSA private key for brokers:
openssl genrsa -out broker.key 2048
where:
Output | Description |
---|---|
|
The private key certificate for the broker. |
3. Generate certificate signing request
Before signing certificates, you must run the following command to generate the broker’s certificate signing request:
openssl req -new -key broker.key -out broker.csr -nodes -config broker.cnf
where:
Inputs | Description |
---|---|
|
Input is a certificate request. Sign and output. |
|
Private key of the CA (created above). |
|
Number of days signed certificates are valid. |
|
Configuration file for CA. |
|
Section in |
|
Broker certificate signing request (CSR generated above). |
Output | Description |
---|---|
|
The signed public key certificate for the broker. |
4. Sign certificates
Sign the certificate with the CA signature:
touch index.txt
echo '01' > serial.txt
openssl ca -config ca.cnf -keyfile ca.key -cert ca.crt -extensions extensions -in broker.csr -out broker.crt -outdir . -batch
chown redpanda:redpanda broker.key broker.crt ca.crt
chmod 400 broker.key broker.crt ca.crt
If generated by a corporate CA, these certificate signing requests must be signed with the following extensions:
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
Configure TLS
To configure TLS, in redpanda.yaml
, enter:
redpanda.yaml
redpanda:
rpc_server_tls: {}
kafka_api:
- address: 0.0.0.0
port: 9092
name: tls_listener
kafka_api_tls:
- name: tls_listener
key_file: broker.key
cert_file: broker.crt
truststore_file: ca.crt
enabled: true
require_client_auth: false
admin_api_tls: []
pandaproxy:
pandaproxy_api_tls: []
schema_registry:
schema_registry_api_tls: []
The following files must be readable by Redpanda, either through 444 permissions
or
Because the keys and certificates are only read at startup, you must restart
Redpanda services after updating systemctl restart redpanda |
To set the RPC port to encrypt replication, add:
redpanda.yaml
redpanda:
rpc_server_tls:
enabled: true
require_client_auth: false
key_file: broker.key
cert_file: broker.crt
truststore_file: ca.crt
Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API. If you configure a TLS listener for the Kafka API, you must add schema_registry_client::broker_tls
and pandaproxy_client::broker_tls
. All APIs, except the internal RPC port, support multiple listeners. See:
Configure mTLS
To enable mTLS, add require_client_auth
set to true
.
For example, for the Kafka API, in redpanda.yaml
, enter:
redpanda.yaml
redpanda:
kafka_api:
- address: 0.0.0.0
port: 9092
name: mtls_listener
kafka_api_tls:
- name: mtls_listener
key_file: mtls_broker.key
cert_file: mtls_broker.crt
truststore_file: mtls_ca.crt
enabled: true
require_client_auth: true
See also: Configure Listeners
Configure mTLS for a Kafka API listener
To enable mTLS for a Kafka API listener, edit redpanda.yaml
:
redpanda.yaml
redpanda:
kafka_api:
- name: internal
address: 0.0.0.0
port: 9092
advertised_kafka_api:
- name: internal
address: <port-clients-connect-to>
port: 9092
kafka_api_tls:
- name: internal
enabled: true
require_client_auth: true
cert_file: <path-to-PEM-formatted-cert-file>
key_file: <path-to-PEM-formatted-key-file>
truststore_file: <path-to-PEM-formatted-CA-file>
|
See also: Configure Listeners
Use rpk with TLS
If you’re using rpk
to interact with the Kafka API using mTLS identity (for example, to manage topics or messages), pass the --tls-key
, --tls-cert
, and --tls-truststore
flags to authenticate.
To interact with the Admin API (for example, to manage users), pass the --admin-api-tls-key
, --admin-api-tls-cert
, and --admin-api-tls-truststore
flags.
rpk topic create test-topic \
--tls-key <path-to-PEM-formatted-key-file> \
--tls-cert <path-to-PEM-formatted-cert-file> \
--tls-truststore <path-to-PEM-formatted-CA-file>
Replace placeholders in brackets.
To check the configuration of the topic, run:
rpk topic describe test-topic <tls-flags-from-above>
By default, rpk
connects to localhost:9092
for Kafka protocol commands. If you’re connecting to a remote broker or if you configured your local broker differently, use the -X brokers=<address:port>
flag.
Monitor TLS certificates
Redpanda exposes several metrics to help administrators manage their installed certificates. When queried, these metrics list details for all resources that have an installed certificate. This may include APIs, storage, or other assets. These metrics also support labels so that you can more readily report statistics on single resources.
Configuring alerts on these metrics is a critical tool for managing certificate expiration and avoiding surprise outages. The public metrics reference contains a full list of available TLS metrics. You can refer to the monitor Redpanda guide for full details on configuring Prometheus to monitor these metrics. This guide also explains how to create a Grafana dashboard for visualizations and alerting.