24.1 beta 23.3 23.2 23.1

Configure Authentication

Different components of Redpanda support different authentication methods. You can configure multiple listeners with redpanda.yaml, and with each listener, you can configure the authentication_method and optionally TLS or mTLS. The guidelines put in place by your organization determine the type of authentication that you use.

See also:

Passwords are in single quotes to avoid conflicts with special characters. Ensure that all values entered are properly enclosed in quotes and escaped as necessary.

Enable authentication

To enable authentication, set kafka_enable_authorization to true, and specify at least one value for the superusers property. This superuser is used to bootstrap permissions for other users in the cluster. See Cluster configuration properties.

To update the kafka_enable_authorization property, run:

rpk cluster config set kafka_enable_authorization true

To specify a superuser, run:

rpk cluster config set superusers ['admin']

To edit a superuser, use the edit command to apply the new value:

rpk cluster config edit

Create superusers

When you configure authentication, you include one or more superusers in the Redpanda configuration file. This user has ALL permissions on the cluster and grants permissions to new users. (Without a superuser, you can create other users, but you can’t grant them permissions to the cluster.)

Specify the name of the superuser. This can be a new user or an existing user. For example, if you use the superuser named admin, then Redpanda allows the admin user to do anything, but Redpanda does not create the admin user. To create this superuser, run:

rpk acl user create <superuser_username> -p '<superuser_password>'

Now the admin user has full access to the cluster and can grant permissions to other users.

The Admin API defaults to localhost:9644. If you’ve configured the Admin API to use a different address/port, use the --api-urls <address:port> flag.

Creating ACLs for users uses the Kafka protocol (default of localhost:9092). Because no ACLs exist at the start, you need a superuser to bypass the requirement of needing ACLs to create ACLs.

To create users and set passwords, run:

rpk acl user create admin \
-p '<password>' \
--api-urls localhost:9644

For information about using rpk to manage ACL users, see rpk acl.

As a security best practice, don’t use the superuser to interact with the cluster, and don’t delete the superuser (in case you need to grant permissions to new users later). In addition, when you create the superuser, you specify a password, which adds a level of security. If you delete the user, someone else could re-create the user with a different password.

SASL/SCRAM

Simple Authentication Security Layer (SASL) is an authentication framework that lets the user choose the authentication mechanism. Redpanda supports the Salted Challenge Response Authentication Mechanism (SCRAM) authentication method.

SASL authentication is only available for the Kafka API.

SASL provides authentication, but not encryption. You can, however, configure TLS to only handle encryption, and use SASL for authentication. This is useful if you require flexibility in your authorization mechanisms.

SCRAM provides strong encryption for user names and passwords by default and does not require an external data store for user information. Redpanda supports the following SASL mechanisms:

  • SCRAM-SHA-256

  • SCRAM-SHA-512

When you run a command with SASL authentication, you must include the mechanism with the following flag:

--sasl-mechanism <mechanism>

For example, to use the SCRAM-SHA-256 mechanism, run:

rpk topic create littlefoot \
--user <username> \
--password <password> \
--sasl-mechanism SCRAM-SHA-256

Configure SASL authentication

To configure SASL authentication for the Kafka API, set authentication_method of the listener to sasl.

In redpanda.yaml, enter:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: sasl_listener
      authentication_method: sasl

Enable SASL with TLS encryption

SASL provides authentication, but not encryption. To provide encryption, you can enable TLS in addition to SASL.

To enable SASL authentication with TLS encryption for the Kafka API, in redpanda.yaml, enter:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: sasl_tls_listener
      authentication_method: sasl
  kafka_api_tls:
    - name: sasl_tls_listener
      key_file: broker.key
      cert_file: broker.crt
      truststore_file: ca.crt
      enabled: true
      require_client_auth: false

Enable Kerberos

Kerberos authentication requires an Enterprise license. To upgrade, contact Redpanda sales.

You configure Kerberos authentication using a keytab, which contains credentials for the service.

Prerequisites

  1. Ensure that host names are fully qualified domain names (FQDN).

  2. Ensure that each broker has a Kerberos configuration file set to use Active Directory or another corporate key distribution center (KDC). Default is at /etc/krb5.conf.

  3. Ensure that the KDC has a valid Kerberos service principal name (SPN) for each broker in the form primary/<FQDN>@<REALM>.

  4. Ensure that each broker has a keytab containing the SPN for that broker. This must be located at an identical file path on each Redpanda broker. Default is /var/lib/redpanda/redpanda.keytab.

For Kerberos (GSSAPI) authentication, Redpanda requires that SASL/SCRAM be enabled for connectivity by rpk, Redpanda Console, and other Redpanda products. Operating with Kerberos only is not a supported configuration.

Enable Kerberos

  1. If the keytab is not in the default location, then set its location:

    rpk cluster config set sasl_kerberos_keytab <path>

  2. If the krb5.conf file is not in the default location, then set its location:

    rpk cluster config set sasl_kerberos_config <path>

  3. Define the primary of the Kerberos SPN to be used by Redpanda with the given keytab. Default is redpanda.

    rpk cluster config set sasl_kerberos_principal <name>

  4. Set sasl_kerberos_principal_mapping. This maps Kerberos user principal names (UPNs) onto Redpanda principals used in the ACLs. For example:

    rpk cluster config set sasl_kerberos_principal_mapping '["RULE:[1:$1@$0](.*@MYDOMAIN.COM)s/@.*//","DEFAULT"]'

    By default, Redpanda matches the primary of the Kerberos UPN of the user. Each rule has the following format:

    • RULE:[n:string](regexp)s/pattern/replacement/g/c where:

      • n is an integer that indicates how many components the target principal should have.

      • If this matches, then a string is formed from string, substituting the realm of the principal for $0 and the n`’th component of the principal for `$n. (For example, if the principal is johndoe/admin@realm.com, then [2:$2$1foo] results in the string adminjohndoefoo.)

      • If this string matches regexp, then the s//[g] substitution command is run over the string.

      • g is optional. It causes the substitution to be global over the string, instead of replacing only the first match in the string.

      • c is optional. It can be either /L or /U to make the match lowercase or uppercase.

    • DEFAULT The principal name is used as the local user name. If the principal has more than one component or is not in the default realm, then the conversion fails.

      Examples of a Kerberos UPN without a host (jdoe@EXAMPLE.COM) and with a host (jdoe/host@EXAMPLE.COM):

      Translation jdoe@EXAMPLE.COM jdoe/host@EXAMPLE.COM

      [1:$1@$0]

      jdoe@EXAMPLE.COM

      Rule does not match[1]

      [1:$1]

      jdoe

      Rule does not match[1]

      [1:$1.foo]

      jdoe.foo

      Rule does not match[1]

      [2:$1/$2@$0]

      Rule does not match[2]

      jdoe/host@EXAMPLE.COM

      [2:$1/$2]

      Rule does not match[2]

      jdoe/host

      [2:$1@$0]

      Rule does not match[2]

      jdoe@EXAMPLE.COM

      [2:$1]

      Rule does not match[2]

      jdoe

      DEFAULT

      jdoe

      jdoe

      The first rule that matches is used to extract a principal.

  5. Append the list of allowed SASL mechanisms that clients can use to authenticate against the Kafka API.

    To get the list of all allowed SASL mechanisms, run:

    rpk cluster config get sasl_mechanisms

    To add support for Kerberos, append the sasl_mechanisms property with the value GSSAPI:

    rpk cluster config set sasl_mechanisms '["SCRAM","GSSAPI"]'

  6. Enable SASL, if not already enabled:

    rpk cluster config set kafka_enable_authorization true
If you’re configuring authentication for the first time, you may need to configure ACLs before users can access Redpanda resources.

Configure Schema Registry and HTTP Proxy to connect to Redpanda with SASL

Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API. For the Kafka username and password, Redpanda uses ephemeral credentials internal to the cluster. Ephemeral credentials are regular SCRAM credentials, but they’re only stored in memory and are lost when a broker restarts. When the Schema Registry or HTTP Proxy start up, they broadcast an ephemeral credential to other brokers over the internal RPC. If authentication fails to a particular broker, new ephemeral credentials are sent to that broker, and the service reconnects.

Schema Registry and HTTP Proxy support SASL/SCRAM but do not support SASL/GSSAPI.

Manual configuration

You can override the ephemeral credentials and manually configure Schema Registry and HTTP Proxy to connect to Redpanda with SASL.

In redpanda.yaml, for schema_registry_client, add:

schema_registry_client:
  brokers:
    - address: 127.0.0.1
      port: 9092
  scram_username: <username>
  scram_password: <password>
  sasl_mechanism: SCRAM-SHA-256

If TLS is in use, additional configuration is required:

schema_registry_client:
  brokers:
    - address: 127.0.0.1
      port: 9092
  broker_tls:
    key_file: broker.key
    cert_file: broker.crt
    truststore_file: ca.crt
    enabled: true
  scram_username: <username>
  scram_password: <password>
  sasl_mechanism: SCRAM-SHA-256

HTTP Proxy has a similar configuration, but in redpanda.yaml, for pandaproxy_client, add:

pandaproxy_client:
  brokers:
    - address: 127.0.0.1
      port: 9092
  broker_tls:
    key_file: broker.key
    cert_file: broker.crt
    truststore_file: ca.crt
    enabled: true
  scram_username: <username>
  scram_password: <password>
  sasl_mechanism: SCRAM-SHA-256

Connect to Redpanda

You can use the newly-created user to interact with Redpanda with rpk:

rpk topic describe test-topic \
--user admin \
--password <password> \
--sasl-mechanism SCRAM-SHA-256 \
--brokers localhost:9092
SUMMARY
=======
NAME        test-topic
PARTITIONS  1
REPLICAS    1

CONFIGS
=======
KEY                     VALUE       SOURCE
cleanup.policy          delete      DYNAMIC_TOPIC_CONFIG
compression.type        producer    DEFAULT_CONFIG
message.timestamp.type  CreateTime  DEFAULT_CONFIG
...
rpk supports SASL/SCRAM, but not SASL/GSSAPI. To create a SASL/SCRAM user, see User create.

Configure basic authentication

Basic authentication is supported on the Admin API, Schema Registry, and HTTP Proxy.

To configure basic authentication on the Admin API, set admin_api_require_auth to true. Administrators create users with rpk acl user create. This adds users to the Redpanda credential store that HTTP basic authentication uses.

You can enable basic authentication to use Kafka API username/password credentials to authenticate to HTTP Proxy and Schema Registry. This requires that SASL is turned on for Kafka API endpoints.

  • HTTP Proxy: Access to the Kafka API impersonates the user whose credentials were used to authenticate to HTTP Proxy, and the user is subject to authorization restrictions by Redpanda ACLs. To support this design, Redpanda passes the username/password in memory to a SASL-enabled Kafka client.

  • Schema Registry: Authorization is "all or nothing": if the user presents a valid user account, then they have full read/write access.

To configure basic authentication, set authentication_method to http_basic.

In redpanda.yaml, enter:

pandaproxy:
  pandaproxy_api:
  - address: "localhost"
    port: 8082
    authentication_method: http_basic

schema_registry:
  schema_registry_api:
    address: "localhost"
    port: 8081
    authentication_method: http_basic

Then to use basic authentication:

rpk acl user create foo --password 'bar' # Creates SASL user "foo" for the Kafka API
curl -u "foo:bar" "http://localhost:8082/topics" # A request to the HTTP Proxy with user foo. Don't forget the colon!
curl -u "foo:bar" "http://localhost:8081/subjects" # A request to the Schema Registry with user foo. Don't forget the colon!

Configure mTLS with authentication

For mTLS authentication, Redpanda uses configurable rules to extract the principal from the Distinguished Name (DN) of an mTLS (X.509) certificate. It uses the principal as the identity or user name.

To enable mTLS authentication, set authentication_method to mtls_identity.

In redpanda.yaml, enter:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: mtls_listener
      authentication_method: mtls_identity
  kafka_api_tls:
    - name: mtls_listener
      key_file: mtls_broker.key
      cert_file: mtls_broker.crt
      truststore_file: mtls_ca.crt
      enabled: true
      require_client_auth: true

By default, Redpanda matches the entire DN. To override the default, specify kafka_mtls_principal_mapping_rules. This is a list of rules that provide a mapping from DN to principal.

Each rule has the following format: RULE:pattern/replacement/[LU]. Where:

  • pattern is a regular expression. For example, to extract the CN field: .*CN=([^,]+).*.

  • replace is used to adjust the match. For example, to use just the first match, use: $1.

  • L makes the match lowercase (optional).

  • U makes the match uppercase (optional).

For example, with the DN: CN=www.redpanda.com,O=Redpanda,OU=Engineering,L=London,S=England,C=UK

Rule Principal

RULE:.*CN=([^,]+).*/$1/

www.redpanda.com

RULE:.*O=([^,]+).*/$1/

Redpanda

RULE:.*O=([^,]+).*/$1/L

redpanda

RULE:.*O=([^,]+),OU=([^,]+),.*,C=([^,]+)/$1-$2-$3/L

redpanda-engineering-uk

DEFAULT

CN=www.redpanda.com,O=Redpanda,OU=Engineering,L=London,S=England,C=UK

The first rule that matches is used to extract a principal.

To update the kafka_mtls_principal_mapping_rules property, run:

rpk cluster config set kafka_mtls_principal_mapping_rules '["DEFAULT"]'

Configure Schema Registry and HTTP Proxy to connect to Redpanda with mTLS

Schema Registry and HTTP Proxy require valid client certificates to secure the connection to Redpanda. Continuing with the previous example, where the certificate contains an identity for authentication (kafka_api listener set to mtls_identity), the following example shows how to connect Schema Registry and HTTP Proxy to Redpanda with mTLS certificate-based identity.

In redpanda.yaml, enter:

schema_registry_client:
  brokers:
    - address: 127.0.0.1
      port: 9092
  broker_tls:
    key_file: schema_registry.key
    cert_file: schema_registry.crt
    truststore_file: ca.crt
    enabled: true
pandaproxy_client:
  brokers:
    - address: 127.0.0.1
      port: 9092
  broker_tls:
    key_file: pandaproxy.key
    cert_file: pandaproxy.crt
    truststore_file: ca.crt
    enabled: true

Disable authentication

To disable authentication for a listener, set authentication_method to none:

pandaproxy:
  pandaproxy_api:
  - address: "localhost"
    port: 8082
    authentication_method: none

schema_registry:
  schema_registry_api:
    address: "localhost"
    port: 8081
    authentication_method: none

If authorization is enabled, connections to this listener use the anonymous user.

To disable authentication on the Kafka API, disable kafka_enable_authorization and set authentication_method to none for all listeners.

For example, run rpk cluster config set kafka_enable_authorization false, and set the following:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: sasl_listener
      authentication_method: none
1. Rule does not match because there are two components in principal name jdoe/host@EXAMPLE.COM.
2. Rule does not match because there is one component in principal name jdoe@EXAMPLE.COM.