Skip to main content
Version: 21.11

Encryption, authorization, and authentication

By default, a Redpanda install starts with no security enabled. Any client can talk over plaintext to Redpanda and perform any operation. You can protect your Redpanda cluster externally through network ACLs, private networks, and so on; this guide does not cover external protection. To ensure defense in depth, it is recommended to always enable encryption and to always use authorization & authentication.

All concepts described here are compatible with Kafka® and its client libraries and CLIs.

Prerequisites​

  • A running Redpanda node.

You can follow our Getting Started guides to learn how to get started with Redpanda.

  • TLS certificates.

TLS certificates are needed for encryption. You do not need TLS for testing, but some concepts below rely on TLS certificates. If you have your own certificates, either self-signed or issued by a trusted Certificate Authority, you can use them for this guide. Otherwise, feel free to use our prepared script to generate certificates:

bash <(curl -s https://gist.githubusercontent.com/0x5d/56422a0c447e58d8ccbfa0ce1fd6bac6/raw/933ca97702f6b844f706b674133105a30bdad3ff/generate-certs.sh)

Encryption: TLS & mTLS​

TLS encrypts network traffic between a client and a server, but does not protect against access to a server. With TLS alone (not mTLS), you need to use ACLs to gate access to your brokers.

mTLS, or 2-way TLS, is an authentication method in which the client and the server must both present certificates. This way both can decide whether the other party can be trusted by ensuring they are who they claim to be.

Cloudfare made a very good description of what mTLS is. If you want you can check their website for more info on that.

If you're on Kubernetes, we wrote a TLS guide exclusive for it.

To enable TLS, set the require_client_auth field to true in the required listener's configuration. For example, to enable mTLS for the "external" API listener:

redpanda:

kafka_api:
# The listener declaration. `name` can have any value.
- name: internal
address: <private IP>
port: 9092

advertised_kafka_api:
# The advertised listeners. `name` should match the name of a declared listener.
# The address:port here is what clients will connect to.
- name: internal
address: localhost
port: 9092

kafka_api_tls:
# The listener's TLS config. `name` must match the corresponding listener's name.
- name: internal
enabled: true
require_client_auth: true # <- This needs to be enabled!
cert_file: <path to PEM-formatted cert file>
key_file: <path to PEM-formatted key file>
truststore_file: <path to PEM-formatted CA file>

If you're using rpk to interact with the Kafka API (managing topics or messages), pass the --tls-key, --tls-cert, and --tls-truststore flags to authenticate.

If you're interacting with the Admin API (managing users for example), pass the --admin-api-tls-key, --admin-api-tls-cert, and --admin-api-tls-truststore flags.

 rpk topic create test-topic \
--tls-key <path to PEM-formatted key file> \
--tls-cert <path to PEM-formatted cert file> \
--tls-truststore <path to PEM-formatted CA file>

The result will be:

TOPIC       STATUS
test-topic OK

And now, check the configuration of the topic with:

rpk topic describe test-topic <tls flags from above>

rpk defaults to connecting to localhost:9092 for the Kafka protocol commands. If you are connecting to a remote broker or have configured your local broker differently, use the --brokers <address:port> flag. A further walkthrough of connecting with TLS and mTLS can be seen on this page.

ACLs​

Access Control Lists (ACLs) allow you to provision users and provide each user fine grained access to specific resources. Redpanda provides ACLs through SASL/SCRAM, a secure and well defined authentication flow.

Enabling SASL/SCRAM​

To enable SASL/SCRAM, set redpanda.enable_sasl to true in the Redpanda configuration file and specify at least one "super user" that will be used to bootstrap permissions for other users in the cluster.

Your config should look something like this:

redpanda:
enable_sasl: true
superusers:
- admin
# The rest of the config...
You can modify the configuration file at `etc/redpanda/redpanda.yaml`. After you modify the file, restart Redpanda for your changes to take effect.

Users​

The superusers section in your redpanda.yaml is the user that you will use to bootstrap other users and ACLs.

For example, if you used super user admin, Redpanda allows the admin user to do anything, but Redpanda does not create the admin user. You must now create this super user.

Creating users happens through the Redpanda admin API, which defaults to localhost:9644. In the rpk commands below, if you have configured the admin API to use a different address/port, use the --api-urls <address:port> flag.

The reason you need a super user is because creating ACLs for users uses the Kafka protocol (default of localhost:9092, mentioned above).

Since no ACLs exist at the start, you need a super user to bypass the requirement of needing ACLs to create ACLs.

You can create users and set passwords by running the following command:

rpk acl user create admin \
-p <password> \
--api-urls localhost:9644

Connect to Redpanda​

You can use the newly created user to interact with Redpanda with the Kafka protocol:

rpk topic describe test-topic \
--user admin \
--password <password> \
--sasl-mechanism SCRAM-SHA-256 \
--brokers localhost:9092
SUMMARY
=======
NAME test-topic
PARTITIONS 1
REPLICAS 1

CONFIGS
=======
KEY VALUE SOURCE
cleanup.policy delete DYNAMIC_TOPIC_CONFIG
compression.type producer DEFAULT_CONFIG
message.timestamp.type CreateTime DEFAULT_CONFIG
...

Authorization​

While authentication tells who you are, authorization tells you what can you do.

Access Control Lists (ACLs)​

Access Control Lists (ACLs) is the main mechanism supported by Redpanda to manage user permissions.

Redpanda stores ACLs internally, replicated with Raft to provide the same consensus guarantees as your data. You can manage your ACLs with rpk acl.

Once you activate SASL, by default, only the super users will have access to the resources. It's recommended to create other users to effectively use Redpanda and then, create ACLs for them.

ACL Terminology​

Entities accessing the resources are called principals. A User:foo is the principal for user "foo".

You can decide whether to to allow or deny permissions to access to the resources.

You can also specify from which hosts they will be allowed or denied access.

This access is represented as operations, such as read, write, or describe, and said operations can be performed on resources, such as a topic. You can filter the resources by name.

Diving deeper​

ACL commands work on a multiplicative basis.

When you're creating if you set up two principals and two permissions the result will be four ACLs: both permissions for the first principal, as well as both permissions for the second principal.

Adding two resources further doubles the ACLs created.

It is recommended to be as specific as possible when granting ACLs.

Granting more ACLs than necessary per principal may inadvertently allow clients to do things they should not, such as deleting topics or joining the wrong consumer group.

Principals​

All ACLs require a principal.

A principal is composed of two parts: the type and the name.

Within Redpanda, currently only one type is supported, "User". In the future Redpanda might add support for authorizing by Group or anything else.

When you create user "bar", Redpanda expects you to add ACLs for "User:bar".

The --allow-principal and --deny-principal flags add this prefix for you if necessary.

The special name '*' matches any name, meaning an ACL with principal "User:*" grants or denies the permission for any user.

Hosts​

Hosts can be seen as an extension of the principal, and effectively gate where the principal can connect from.

When creating ACLs, unless otherwise specified, the default host is the wildcard * which allows or denies the principal from all hosts.

If specifying hosts, you must pair the --allow-host flag with the --allow-principal flag

The same applies to the --deny-host flag with the --deny-principal flag.

Resources​

A resource is what an ACL allows or denies access to.

There are four resources within Redpanda: topics, groups, cluster, and transactionalid.

Names for each of these resources can be specified with their respective flags.

Resources combine with the operation that is allowed or denied on that resource.

By default, resources are specified on an exact name match (a "literal" match).

The --resource-pattern-type flag can be used to specify that a resource name is "prefixed", meaning to allow anything with the given prefix.

A literal name of "foo" will match only the topic "foo", while the prefixed name of "foo-" will match both "foo-bar" and "foo-jazz".

The special wildcard resource name '*' matches any name of the given resource type (--topic '*' matches all topics).

Operations​

Pairing with resources, operations are the actions that are allowed or denied. Redpanda has the following operations:

Operation Description
 ALL Allows all operations below.
READ Allows reading a given resource.
WRITE
 Allows writing to a given resource.
CREATE
 Allows creating a given resource.
DELETE
 Allows deleting a given resource
ALTER
 Allows altering non-configurations.
DESCRIBE Allows querying non-configurations.
DESCRIBE_CONFIGS Allows describing configurations.
ALTER_CONFIGS
 Allows altering configurations.

Producing/Consuming​

The following lists the operations needed for each individual client request, where resource corresponds to the resource flag, and "for xyz" corresponds to the resource name(s) in the request:

The following lists the operations needed for each individual client request,
where "on RESOURCE" corresponds to the resource flag, and "for xyz" corresponds
to the resource name(s) in the request:

PRODUCING/CONSUMING

Produce WRITE on TOPIC for topics
WRITE on TRANSACTIONAL_ID for the transaction.id

Fetch READ on TOPIC for topics

ListOffsets DESCRIBE on TOPIC for topics

Metadata DESCRIBE on TOPIC for topics
CREATE on CLUSTER for kafka-cluster (if automatically creating topics)
CREATE on TOPIC for topics (if automatically creating topics)

OffsetForLeaderEpoch DESCRIBE on TOPIC for topics

GROUP CONSUMING

FindCoordinator DESCRIBE on GROUP for group
DESCRIBE on TRANSACTIONAL_ID for transactional.id (transactions)

OffsetCommit READ on GROUP for groups
READ on TOPIC for topics

OffsetFetch DESCRIBE on GROUP for groups
DESCRIBE on TOPIC for topics

OffsetDelete DELETE on GROUP for groups
READ on TOPIC for topics

JoinGroup READ on GROUP for group
Heartbeat READ on GROUP for group
LeaveGroup READ on GROUP for group
SyncGroup READ on GROUP for group

TRANSACTIONS (including FindCoordinator above)

AddPartitionsToTxn WRITE on TRANSACTIONAL_ID for transactional.id
WRITE on TOPIC for topics

AddOffsetsToTxn WRITE on TRANSACTIONAL_ID for transactional.id
READ on GROUP for group

EndTxn WRITE on TRANSACTIONAL_ID for transactional.id

TxnOffsetCommit WRITE on TRANSACTIONAL_ID for transactional.id
READ on GROUP for group
READ on TOPIC for topics

ADMIN

CreateTopics CREATE on CLUSTER for kafka-cluster
CREATE on TOPIC for topics
DESCRIBE_CONFIGS on TOPIC for topics, for returning topic configs on create

CreatePartitions ALTER on TOPIC for topics

DeleteTopics DELETE on TOPIC for topics
DESCRIBE on TOPIC for topics, if deleting by topic id (in addition to prior ACL)

DeleteRecords DELETE on TOPIC for topics

DescribeGroup DESCRIBE on GROUP for groups

ListGroups DESCRIBE on GROUP for groups
or, DESCRIBE on CLUSTER for kafka-cluster

DeleteGroups DELETE on GROUP for groups

DescribeConfigs DESCRIBE_CONFIGS on CLUSTER for cluster (broker describing)
DESCRIBE_CONFIGS on TOPIC for topics (topic describing)

AlterConfigs ALTER_CONFIGS on CLUSTER for cluster (broker altering)
ALTER_CONFIGS on TOPIC for topics (topic altering)

You can also get this information at the CLI by running:

rpk acl --help-operations 

In flag form to set up a general producing/consuming client, you can invoke rpk acl create up to three times with the following (including your --allow-principal):

--operation write,read,describe --topic [topics]
--operation describe,read --group [group.id]
--operation describe,write --transactional-id [transactional.id]

Permissions​

A client can be allowed access or denied access. By default, all permissions are denied.

You only need to specifically deny a permission if you allow a wide set of permissions and then want to deny a specific permission in that set.

You could allow all operations, and then specifically deny writing to topics.

Management​

Creating ACLs works on a specific ACL basis, but listing and deleting ACLs works on filters.

Filters allow matching many ACLs to be printed listed and deleted at once.

Because this can be risky for deleting, the delete command prompts for confirmation by default.

RPK ACL & Management of users​

rpk acl is a command made to both manage your ACLs as well as your SASL users.

If you're on Kubernetes you can use kubectl exec to run RPK's commands.

Here's the general usage:

rpk acl [command] [flags]

For example, to create an user:

rpk acl user create \
--new-username Jack \
--new-password <password> \
--api-urls localhost:9644

Results in:

Created user 'Jack'

Here are all the available commands and how they interact with Redpanda:

CommandProtocolDefault Port
userAdmin API9644
listKafka API9092
createKafka API9092
deleteKafka API9092

You can always run rpk acl -h to get more information.

Global Flags​

Every rpk acl command can have these flags:

FlagDescription
--admin-api-tls-certThe certificate to be used for TLS authentication with the Admin API.
--admin-api-tls-enabledEnable TLS for the Admin API (not necessary if specifying custom certs). This is assumed as true when passing other --admin-api-tls flags.
--admin-api-tls-keyThe certificate key to be used for TLS authentication with the Admin API.
--admin-api-tls-truststoreThe truststore to be used for TLS communication with the Admin API.
--brokersComma-separated list of broker ip:port pairs (e.g. --brokers '192.168.78.34:9092,192.168.78.35:9092,192.179.23.54:9092' ). Alternatively, you may set the REDPANDA_BROKERS environment variable with the comma-separated list of broker addresses.
--configRedpanda config file, if not set the file will be searched for in the default locations
-h, --helpHelp for acl.
--passwordSASL password to be used for authentication.
--sasl-mechanismThe authentication mechanism to use. Supported values: SCRAM-SHA-256, SCRAM-SHA-512.
--tls-certThe certificate to be used for TLS authentication with the broker.
--tls-enabledEnable TLS for the Kafka API (not necessary if specifying custom certs).This is assumed as true when passing other --tls flags.
--tls-keyThe certificate key to be used for TLS authentication with the broker.
--tls-truststoreThe truststore to be used for TLS communication with the broker.
--userSASL user to be used for authentication.

Create ACLs​

Following the multiplying effect of combining flags, the create command works on a straightforward basis: every ACL combination is a created ACL.

At least one principal, one host, one resource, and one operation is required to create a single ACL.

Here's the general usage:

rpk acl create/delete [globalACLFlags] [localFlags]

You can use the global flags that we saw before and some other local flags. Here's the available local flags:

FlagDescription
--allow-hostHost for which access will be granted. (repeatable)
--allow-principalPrincipals to which permissions will be granted. (repeatable)
--clusterWhether to grant ACLs to the cluster.
--deny-hostHost from which access will be denied. (repeatable)
--deny-principalPrincipal to which permissions will be denied. (repeatable)
--groupGroup to grant ACLs for. (repeatable)
-h, --helpHelp for create.
--name-patternThe name pattern type to be used when matching the resource names.
--operationOperation that the principal will be allowed or denied. Can be passed many times.
--resource-pattern-typePattern to use when matching resource names (literal or prefixed) (default "literal")
--topicTopic to grant ACLs for. (repeatable)
--transactional-idTransactional IDs to grant ACLs for. (repeatable)

Examples:

Allow all permissions to user bar on topic "foo" and group "g":

rpk acl create --allow-principal bar --operation all --topic foo --group g

Allow read permissions to all users on topics biz and baz:

rpk acl create --allow-principal '*' --operation read --topic biz,baz

Allow write permissions to user buzz to transactional id "txn":

rpk acl create --allow-principal User:buzz --operation write --transactional-id txn

List/Delete ACLs​

List and Delete work in a similar multiplying effect as creating ACLs, but delete is more advanced.

They work on a filter basis. Any unspecified flag defaults to matching everything (all operations, or all allowed principals, etc).

To ensure that you do not accidentally delete more than you intend, this command prints everything that matches your input filters and prompts for a confirmation before the delete request is issued.

Anything matching more than 10 ACLs is going to ask again for confirmation.

If no resources are specified, all resources are matched. If no operations are specified, all operations are matched.

You can also opt in to matching everything. --operation any matches any operation, for example.

The --resource-pattern-type, defaulting to any, configures how to filter resource names:

  • any returns exact name matches of either prefixed or literal pattern type
  • match returns wildcard matches, prefix patterns that match your input, and literal matches
  • prefix returns prefix patterns that match your input (prefix "fo" matches "foo")
  • literal returns exact name matches

Here's the general usage:

rpk acl list/delete [globalACLFlags] [localFlags]

You can use the global flags that we saw before and some other local flags. Here's the available local flags:

 FlagDescription 
--allow-host Allowed host ACLs to list/remove. (repeatable)
 --allow-principalAllowed principal ACLs to list/remove. (repeatable)
 --clusterWhether to list/remove ACLs to the cluster.
 --deny-hostDenied host ACLs to list/remove. (repeatable)
 --deny-principalDenied principal ACLs to list/remove. (repeatable)
 -d, --dry Dry run: validate what would be deleted.
 --groupGroup to list/remove ACLs for. (repeatable)
 -h, --helpHelp for delete.
 --no-confirm Disable confirmation prompt.
 --operationOperation to list/remove. (repeatable)
 -f, --print-filtersPrint the filters that were requested. (failed filters are always printed)
--resource-pattern-typePattern to use when matching resource names. (any, match, literal, or prefixed) (default "any")
--topicTopic to list/remove ACLs for. (repeatable)
--transactional-id Transactional IDs to list/remove ACLs for. (repeatable)

User​

Used to manage the SASL users.

If SASL is enabled, a SASL user is what you use to talk to Redpanda, and ACLs control what your user has access to.

Using SASL requires setting "enable_sasl: true" in the redpanda section of your redpanda.yaml.

Here's the general usage:

rpk acl user [command] [globalACLFlags] [globalUserFlags]

For users you can use the global flags that we saw before and these global user flags. Here's the available global user flags:

FlagDescriptionSupported Value
--api-urlsThe comma-separated list of Admin API addresses (IP:port). You must specify one for each node.strings
-h, --helpHelp for user.
User create​

This command creates a single SASL user with the given password, optionally with a custom "mechanism".

SASL consists of three parts: a username, a password, and a mechanism.

The mechanism determines which authentication flow the client will use for this user/pass.

Redpanda currently supports two mechanisms: SCRAM-SHA-256, the default, and SCRAM-SHA-512, which is the same flow but uses sha512 rather than sha256.

Before a created SASL account can be used, you must also create ACLs to grant the account access to certain resources in your cluster.

Here's the general usage:

rpk acl user create [USER] -p [PASSWORD] [globalACLFlags] [globalUserFlags] [localFlags]

Here's the local flags:

FlagDescription
-h, --helpHelp for create.
--mechanismSASL mechanism to use (scram-sha-256, scram-sha-512, case insensitive) (default "scram-sha-256")

User delete​

This command deletes the specified SASL account from Redpanda. This does not delete any ACLs that may exist for this user.

You may want to recreate the user later, as well, not all ACLs have users that they describe (instead they are for wildcard users).

Here's the general usage:

rpk acl user delete [USER] [globalACLFlags] [globalUserFlags] 

User list​

Used to list SASL users. Here's the general usage:

rpk acl user list [globalACLFlags] [globalUserFlags]

You can also use the shortened version changing list to ls.