Configuring Redpanda mTLS on Kubernetes

Deprecated red

The Cluster and Console resources are deprecated. For details, see the deprecation notice. To migrate to the Redpanda resource, see Migrate from Cluster and Console Custom Resources

This section provides details on how to enable mutual Transport Layer Security (mTLS) authentication on Kubernetes. mTLS authenticates the server and encrypts communication between the server and client, as one-way TLS does, and also authenticates the client. Redpanda supports mTLS authentication on all four APIs.

For general security information, see Redpanda Security on Kubernetes.

Prerequisites

  1. Configure the kubectl context.

  2. Install cert-manager.

  3. Install the Redpanda operator.

If you haven’t completed these tasks, follow one of the Install Redpanda Guide for Kubernetes to get set up before you configure TLS. Just stop before you install and connect to the Redpanda cluster.

Step 1: Create the cluster specification file

The Redpanda GitHub repository contains an mtls.yaml sample configuration file you can use to enable mTLS authentication: redpanda-examples/docs/example-config/kubernetes/mtls.yaml.

To modify the sample file, save it locally and make the required changes.

The text below is the Redpanda mtls.yaml file with the relevant sections highlighted:

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
  name: cluster-sample-mtls
spec:
  image: "redpandadata/redpanda"
  version: "latest"
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 1.2G
    limits:
      cpu: 1
      memory: 1.2G
  configuration:
    rpcServer:
      port: 33145
    kafkaApi:
     - port: 9092
       tls:
         enabled: true
         requireClientAuth: true
    pandaproxyApi:
     - port: 8082
       tls:
         enabled: true
         requireClientAuth: true
    schemaRegistry:
       port: 8081
       tls:
         enabled: true
         requireClientAuth: true
    adminApi:
     - port: 9644
       tls:
         enabled: true
         requireClientAuth: true
     developerMode: true

Complete these steps to configure the cluster specification file. The steps reference the highlighted lines in the file:

  1. Configure the cluster name. The following property in the file is the name of the cluster. Change this value to the name of your cluster:

    name: cluster-sample-mtls

  2. Enable mTLS for each API individually. You can enable mTLS on one or more of the APIs. In the example, all four APIs have mTLS enabled:

    kafkaApi:
 - port: 9092
   tls:
     enabled: true
     requireClientAuth: true
pandaproxyApi:
 - port: 8082
   tls:
     enabled: true
     requireClientAuth: true
schemaRegistry:
   port: 8081
   tls:
     enabled: true
     requireClientAuth: true
adminApi:
 - port: 9644
   tls:
     enabled: true
     requireClientAuth: true

Step 2: Configure external connectivity (optional)

The table below lists the supported listener configurations for each API with mTLS enabled. See Listeners.

API Listener configurations with mTLS

Kafka API

HTTP Proxy

Admin API

  • One internal listener with mTLS enabled

  • One internal listener and one external listener. Only one of the listeners can have mTLS enabled.

Schema Registry

  • One internal listener with mTLS enabled

  • One listener that is used for internal and external connectivity with mTLS enabled

You can specify up to two listeners for each API, but only one listener can have mTLS enabled. If you do have two listeners, one must be internal, and one must be external. The exception is Schema Registry. The Schema Registry listener can be internal, or it can be an internal port that is used internally and externally. If you enable external connectivity on Schema Registry, the Kubernetes node port connects to the internal Redpanda port to provide external connectivity.

To enable external connectivity with mTLS, add the following lines to each API in the configuration file that you created in Step 1:

 - external:
     enabled: true
     subdomain: <subdomain_name>

The subdomain field lets you specify the advertised address of the external listener. The subdomain addresses, including the brokers, must be registered with a DNS provider, such as Amazon Route 53. You only need to include the subdomain name in this field, not the brokers. Each API in the configuration file must have the same subdomain specified.

The external port is generated automatically, and you do not need to specify it. In the following example, mTLS is enabled on the external listener for the Kafka API. Enable external connectivity the same way for Admin API and HTTP Proxy.

kafkaApi:
 - port: 9092
 - external:
     enabled: true
     subdomain: <subdomain_name>
   tls:
     enabled: true
     requireClientAuth: true

The Schema Registry syntax is slightly different, in that the ports are not a list. You can specify one internal port and one external port. Schema Registry always uses an internal port and with external connectivity configured, the Kubernetes node port connects to the internal Redpanda port. Configure mTLS with external connectivity for Schema Registry like this:

schemaRegistry:
  port: 8081
  external:
    enabled: true
    subdomain: <subdomain_name>
  tls:
    enabled: true
    requireClientAuth: true

For information about external connectivity, including subdomains, see External connectivity.

Step 3: Provide an issuer or certificate (optional)

Kafka API and Schema Registry let you provide a certificate issuer or certificate for the node certificate.

When you enable mTLS, the Redpanda operator generates a root certificate for each API. The root certificate is local to the cluster and the operator uses the root certificate to generate leaf certificates for the nodes and the client. However, for Kafka API and Schema registry you can instead specify a certificate issuer or a certificate.

For information about how certificates are created and used in Redpanda, see Certificates.

Provide an issuer

To provide a certificate issuer, add the issuerRef property to the cluster specification file that you created in the previous step. For information about issuers, see the cert-manager Issuer documentation.

You can provide an issuer for kafkaAPI or schemaRegistry in the same way. The example here is the kafkaAPI configuration configuration with the issuerRef property highlighted:

kafkaApi:
 - port: 9092
   tls:
     enabled: true
     issuerRef:
       name: <issuer_name>
       kind: <issuer>
     requireClientAuth: true

The issuerRef property contains the following variables:

Variable Description

issuer_name

The name of the issuer or cluster issuer.

issuer

A Kubernetes resource that represents a certificate authority. The value of this property can be Issuer or ClusterIssuer. If the kind property is not set, or if it is set to Issuer, an issuer with the name specified in the name property that exists in the same namespace as the certificate is used.

Provide a certificate

You can provide a certificate as a secret by adding the nodeSecretRef property to the cluster specification file that you created. For information about Secrets, see the Kubernetes Secrets documentation. The cert-manager Certificate documentation contains detailed information about certificates, including a diagram of the certificate lifecycle.

You can provide a certificate for kafkaAPI or schemaRegistry in the same way. The following example is the kafkaAPI configuration with the nodeSecretRef property highlighted:

kafkaApi:
 - port: 9092
   tls:
     enabled: true
     nodeSecretRef:
       name: <secret_name>
       namespace: <secret_namespace>
     requireClientAuth: true

The nodeSecretRef property contains the following variables:

Variable Description

secret_name

Name of the certificate secret.

secret_namespace

Kubernetes namespace where the certificate secret is. If the secret is in a different namespace than the Redpanda cluster, the operator copies it to the namespace of the Redpanda cluster.
To include a ca.crt file in the Secret, you must create an Opaque Secret instead of a TLS Secret. See Opaque Secrets in the Kubernetes documentation.

Step 4: Create the Redpanda cluster

After you configure the cluster specification file, you must run the kubectl apply command to create the cluster. You can run the command using a path to the cluster specification file on your local machine, or you can use the URL to the mtls.yaml file.

If you modified the file in the previous step, you have the file saved locally. To create the Redpanda cluster, run:

kubectl apply -f <cluster_specification.yaml>

If you did not modify the example file, you can use the URL to the example file in GitHub to create the cluster:

kubectl apply -f https://raw.githubusercontent.com/redpanda-data/redpanda-examples/main/docs/example-config/kubernetes/mtls.yaml

Step 5: Create the ConfigMap

Create a YAML file to hold the configuration for mTLS, including the location of the public certificate. In the next step, you create the Pod, which consumes this ConfigMap. This lets you run rpk commands with mTLS authentication.

For detailed information, see the Kubernetes ConfigMaps documentation.

  1. Copy the text below and save it locally as a YAML file, such as mtls_config_map.yaml.

    apiVersion: v1
kind: ConfigMap
metadata:
  name: <ConfigMap_name>
data:
  redpanda.yaml: |
    redpanda:
    rpk:
      kafka_api:
        brokers:
          - <cluster_name>-0.<cluster_name>.default.svc.cluster.local:9092
        tls:
          key_file: <key_file_path>/tls.key
          cert_file: <cert_file_path>/tls.crt
          truststore_file: <truststore_file_path>/ca.crt

  2. In the file you just saved, configure these variables:

    Variable Description

    ConfigMap_name

    Name of the ConfigMap. This can be any string. This is what you use to reference the ConfigMap in the next step when you configure the Pod.

    cluster_name

    Name of the Redpanda cluster you defined in the cluster specification file.

    key_file_path

    Directory where you want to mount the tls.key private client key. Generally this is /etc/tls/certs.

    cert_file_path

    Filename and directory where you want to mount the tls.crt private key. Generally this is /etc/tls/certs.

    truststore_file_path

    Directory where you want to mount the ca.crt file. Generally this is /etc/tls/certs/ca.

  3. Save the file.

External connectivity

If you’re configuring mTLS with external connectivity, you must configure the brokers accordingly. Replace the brokers property in the example file with this:

brokers:
 - 0.<subdomain_name>.:<node_port>

Configure the following variables in the brokers property:

Variable Description

subdomain_name

Name of the subdomain that you include in the cluster specification file in Step 1.

node_port

Kafka API external port. Unless you include this in the cluster specification file, this port is autogenerated by Kubernetes.

Step 6: Configure the Pod

The Pod is the process that consumes the ConfigMap you created. This Pod runs the Redpanda image for running rpk, which is part of the Redpanda image.

For detailed information about Pods, see the Kubernetes Pods documentation.

  1. Copy the text below and save it locally as a YAML file, such as mtls_pod.yaml.

    apiVersion: v1
kind: Pod
metadata:
  name: <pod_name>
spec:
  containers:
    - name: rpk
      image: 'redpandadata/redpanda:<redpanda-version>'
      command:
        - /bin/bash
        - '-c'
      args:
        - sleep infinity
      volumeMounts:
        - mountPath: <key_file_path>
          name: <tls_volume_name>
        - mountPath: <truststore_file_path>
          name: <ca_volume_name>
        - mountPath: /etc/redpanda
          name: <rpk_volume_name>
  restartPolicy: Never
  volumes:
    - name: <tls_volume_name>
      secret:
        secretName: <cluster_name>-user-client
    - name: <ca_volume_name>
      secret:
        secretName: <cluster_name>-redpanda
    - name: <rpk_volume_name>
      configMap:
        name: <configMap_name>

  2. In the file you just saved, configure these variables:

    Variable Description

    pod_name

    Name of the Pod. This is the Pod that runs rpk. This can be any string.

    args

    Specifies what you want the Pod to do. You can run rpk commands here. This example uses the sleep infinity argument, which tells the Pod to keep running so that you can run as many rpk commands as you want from the command line.

  3. Configure the volumeMounts properties. There are three: one for tls, one for ca, and one for rpk.

    • tls - The path and the name of the tls.crt and tls.key volume mount.

      Variable Description

      key_file_path

      The same path that you specified in the key_file_path and cert_file_path properties in the ConfigMap. Generally this is /etc/tls/certs.

      tls_volume_name

      Must match the tls_volume_name in the volumeMounts property.

    • ca - The path and the name of the ca.crt volume mount.

      Variable Description

      truststore_file_path

      The same path that you specified in the truststore_file_path property in the ConfigMap. Generally this is /etc/tls/certs/ca.

      ca_volume_name

      This can be any string, but it must match the ca_volume_name in the volumes property of this file.

    • rpk - The path and the name of the rpk volume mount.

      Variable Description

      name

      This can be any string, but it must match the rpk_volume_name in the volumes property of this file.

  4. Configure the volumes properties. There are three: one for tls, one for ca, and one for rpk.

    • tls - The name and secret of the tls.crt and tls.key volume mount.

      Variable Description

      tls_volume_name

      This must be the same as the tls_volume_name in the volumeMounts property of this file.

      cluster_name

      Cluster name that you defined in the cluster specification file in Step 1. The secretName property specifies the name of the client secret. For the Kafka API, this is <cluster_name>-user-client.

    • ca - The name and secret of the ca.crt volume mount.

      Variable Description

      name

      This must be the same as the ca_volume_name in the volumeMounts property.

      cluster_name

      Cluster name that you defined in the cluster specification file in Step 1. The secretName property specifies the name of the node secret. For the Kafka API, this is <cluster_name>-redpanda.

    • rpk - The volume name and ConfigMap name of the rpk volume mount.

      Variable Description

      rpk_volume_name

      This must match the <rpk_volume_name> in the volumeMounts property of this file.

      configMap_name

      ConfigMap name you specified in the name property of the ConfigMap in the previous step.

  5. Configure the <redpanda-version> variable. Add a Redpanda version, such as v21.11.11. You can find all Redpanda version tags in the Redpanda Docker Hub repository.

  6. Save the file.

Step 7: Create the Pod

To create the pod, run:

kubectl apply -f <mtls_pod.yaml>

Step 8: Connect to Redpanda

Now that you have mTLS enabled and the Pod created, you can start using rpk to interact with Redpanda. Note that each time you run an rpk command, rpk establishes a connection and authenticates the server and the server authenticates the client.

  1. Create a topic with this command:

kubectl exec <pod_name> -- rpk topic create <topic_name>

Configure these variables:

Variable Description

pod_name

Name of the Pod that you specified in the Pod configuration file.

topic_name

Name of the topic that you’re creating with this command.

  1. To describe the topic, run:

kubectl exec <pod_name> -- rpk topic describe <topic_name>
You don’t need to specify the brokers in these commands, because they were defined in the ConfigMap. If you include brokers in the rpk commands, it overrides the brokers in the ConfigMap.

Step 9: Clean up

You can use rpk commands to start producing and consuming to your cluster.

When you’re ready to delete your cluster and configuration files, run:

kubectl delete -f <cluster_specification.yaml> -f <mtls_config_map.yaml> -f <mtls_pod.yaml>

Suggested reading