Configure Kafka TLS Encryption

Loading...

By default, Redpanda data is sent unencrypted. A security best practice is to enable encryption with TLS or mTLS.

• Transport Layer Security (TLS), previously SSL, provides encryption for client-server communication. A server certificate prevents third parties from accessing data transferred between the client and server.
• mTLS, or 2-way TLS, is a protocol that authenticates both the server and the client. In addition to the server certificate required in TLS, mTLS also requires the client to give a certificate. This involves more overhead to implement, but it can be useful for environments that require additional security and only have a small number of verified clients.

For each Redpanda broker, specify the key file (broker.key), the certificate file (broker.crt), and the truststore file (ca.crt). Each broker has its own broker.key and broker.crt, but all brokers can have the same ca.crt.

Prerequisites

TLS certificates are necessary for encryption. You can use your own certificates, either self-signed or issued by a trusted certificate authority (CA).

Create certificate signing requests

This step creates the certificate signing request for the CA to extend trust over the broker's certificates.

1. Create broker configuration file (broker.cnf)

A subject alternative name (SAN) indicates all of the domain names and IP addresses that are secured by the certificate. Depending on the address the client uses to connect to Redpanda, you might need to create a CNF file for each broker to modify the alt_names section with organizational details. For production usage, edit alt_names with DNS resolutions and/or the IP addresses.

broker.cnf

[ req ]
prompt = no
distinguished_name = distinguished_name
req_extensions     = extensions

[ distinguished_name ]
organizationName = Redpanda

[ extensions ]
keyUsage         = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
subjectAltName   = @alt_names

[ alt_names ]
DNS.1 = localhost
DNS.2 = redpanda
DNS.3 = console
DNS.4 = connect
IP.1  = 127.0.0.1

You could configure alternative names with a single version of broker.key/broker.crt, as long as you update the certificate for all brokers in the cluster any time you edit an entry. For example:

[ alt_names ]
DNS.1 = broker1.example.com
DNS.2 = broker2.example.com
DNS.3 = broker3.example.com

2. Generate broker private key (broker.key)

Generate a 2048-bit RSA private key for brokers:

openssl genrsa -out broker.key 2048

where:

Output	Description
-out broker.key	The public key certificate of the CA.

3. Create a certificate signing request (broker.csr)

Create a public certificate signing request (CSR) for the brokers:

openssl req -new -config broker.cnf -key broker.key -out broker.csr

where:

Inputs	Description
-new	New request.
-config broker.cnf	Configuration file to use when generating certificates (created above).
-key broker.key	Private key of the broker (created above).

Output	Description
-out broker.csr	The public key certificate of the CA.

If generated by a corporate CA, these certificate signing requests must be signed with the following extensions:

keyUsage         = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth

Create a local CA for self-signed certificates

note

This step is required if you want to generate multiple certificates all signed by the same root; for example, you want to use mTLS but issue different certificates to multiple Redpanda brokers and clients.

To generate a self-signed certificate in a single command:

openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout broker.key -out broker.crt -subj "/CN=redpanda" -addext "subjectAltName = DNS:localhost, IP: 127.0.0.1"

1. Create CA configuration file (ca.cnf)

Edit the distinguished_name section with your own organization details. For default_md, sha256 is the minimum message digest level. The subjectAltName must be accurate for the broker's certificate.

ca.cnf

[ ca ]
default_ca = CA_default

[ CA_default ]
default_days    = 365
default_md      = sha256
unique_subject  = no

[ req ]
prompt = no
distinguished_name = distinguished_name
x509_extensions    = extensions

[ distinguished_name ]
organizationName = Redpanda
commonName       = Redpanda CA

[ extensions ]
keyUsage         = critical,digitalSignature,nonRepudiation,keyEncipherment,keyCertSign
basicConstraints = critical,CA:true,pathlen:1
subjectAltName   = @alt_names

[ alt_names ]
DNS.1 = localhost
IP.1 = 127.0.0.1

2. Generate CA private key and public certificate

Create a CA key to self-sign certificates:

openssl genrsa -out ca.key 2048
chmod 400 ca.key

Create a public certificate for the CA:

openssl req -new -x509 -config ca.cnf -key ca.key -days 365 -batch -out ca.crt

where:

Inputs	Description
-new	New request.
-x509	Create an X.509 certificate, instead of a certificate signing request (CSR).
-config ca.cnf	Configuration file to use when generating certificates (created above).
-key ca.key	Private key of the CA (created above).
-days 365	Number of days signed certificates are valid.
-batch	Batch mode, where certificates are certified automatically.

Output	Description
-out ca.crt	The public key certificate of the CA.

3. Sign certificates

Sign and generate an X.509 certificate for the Redpanda broker:

openssl x509 -req -signkey ca.key -days 365 -extfile broker.cnf -extensions extensions -in broker.csr -out broker.crt

where:

Inputs	Description
-req	Input is a certificate request. Sign and output.
-signkey ca.key	Private key of the CA (created above).
-days 365	Number of days signed certificates are valid.
-extfile broker.cnf	Configuration file for CA.
-extensions extensions	Section in broker.cnf to use when applying extensions.
-in broker.csr	Broker certificate signing request (CSR generated above).

Output	Description
-out broker.crt	The signed public key certificate for the broker.

Configure TLS

To configure TLS, in redpanda.yaml, enter:

redpanda.yaml

redpanda:
  rpc_server_tls: {}
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: tls_listener
  kafka_api_tls:
    - name: tls_listener
      key_file: broker.key
      cert_file: broker.crt
      truststore_file: ca.crt
      enabled: true
      require_client_auth: false
  admin_api_tls: []
pandaproxy:
  pandaproxy_api_tls: []
schema_registry:
  schema_registry_api_tls: []

To set the RPC port to encrypt replication, add:

redpanda.yaml

redpanda:
  rpc_server_tls:
    enabled: true
    require_client_auth: false
    key_file: broker.key
    cert_file: broker.crt
    truststore_file: ca.crt

Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API. If you configure a TLS listener for the Kafka API, you must add schema_registry_client::broker_tls and pandaproxy_client::broker_tls. All APIs, except the internal RPC port, support multiple listeners. See:

• Configure Schema Registry and HTTP Proxy to connect to Redpanda with SASL
• Configure Listeners

Configure mTLS

To enable mTLS, add require_client_auth set to true.

For example, for the Kafka API, in redpanda.yaml, enter:

redpanda.yaml

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: mtls_listener
  kafka_api_tls:
    - name: mtls_listener
      key_file: mtls_broker.key
      cert_file: mtls_broker.crt
      truststore_file: mtls_ca.crt
      enabled: true
      require_client_auth: true

See also: Configure Listeners

Configure mTLS for a Kafka API listener

To enable mTLS for a Kafka API listener, edit redpanda.yaml:

redpanda.yaml

redpanda:
  kafka_api:
  - name: internal
    address: 0.0.0.0
    port: 9092

  advertised_kafka_api:
  - name: internal
    address: <port-clients-connect-to>
    port: 9092

  kafka_api_tls:
  - name: internal
    enabled: true
    require_client_auth: true
    cert_file: <path-to-PEM-formatted-cert-file>
    key_file: <path-to-PEM-formatted-key-file>
    truststore_file: <path-to-PEM-formatted-CA-file>

notes

• Remember to replace placeholders in brackets.
• kafka_api is the listener declaration. This name can have any value.
• advertised_kafka_api is the advertised listener. This name should match the name of a declared listener. This address is the host name clients use to connect to the broker.
• kafka_api_tls is the listener's TLS configuration. This name must match the corresponding listener's name.

See also: Configure Listeners

Use rpk with TLS

If you're using rpk to interact with the Kafka API using mTLS identity (for example, to manage topics or messages), pass the --tls-key, --tls-cert, and --tls-truststore flags to authenticate.

To interact with the Admin API (for example, to manage users), pass the --admin-api-tls-key, --admin-api-tls-cert, and --admin-api-tls-truststore flags.

 rpk topic create test-topic \
--tls-key <path-to-PEM-formatted-key-file> \
--tls-cert <path-to-PEM-formatted-cert-file> \
--tls-truststore <path-to-PEM-formatted-CA-file>

note

Remember to replace placeholders in brackets.

Example output

TOPIC       STATUS
test-topic  OK

To check the configuration of the topic, run:

rpk topic describe test-topic <tls-flags-from-above>

By default, rpk connects to localhost:9092 for Kafka protocol commands. If you're connecting to a remote broker or if you configured your local broker differently, use the --brokers <address:port> flag.

Suggested reading

• TLS configuration for Redpanda and rpk
• Work with schema registry