Configure Authorization

Authorization works in tandem with authentication. Access-control lists (ACLs) provide a way to configure fine-grained access to provisioned users. ACLs work with SASL/SCRAM and with mTLS with principal mapping for authentication.

ACLs

Access-control lists (ACLs) are the primary mechanism used by Redpanda to manage user permissions. Redpanda stores ACLs internally, replicated with Raft to provide the same consensus guarantees as your data. You can manage ACLs with rpk acl.

After you enable authorization, by default, only superusers have access to the resources. Redpanda recommends creating other users to effectively use Redpanda and then, create ACLs for them.

ACL terminology

Entities accessing the resources are called principals. A User:foo is the principal for user "foo". You can decide whether to allow or deny permissions to access to the resources.

You can also specify the hosts for which they have access.

This access is represented as operations, such as read, write, or describe, and the operations can be performed on resources, such as a topic. You can filter the resources by name.

ACL commands work on a multiplicative basis. If you set up two principals and two permissions, the result is four ACLs: both permissions for the first principal, as well as both permissions for the second principal. Adding two resources further doubles the ACLs created.

Be as specific as possible when granting ACLs.

Granting more ACLs than necessary per principal may inadvertently allow clients to do things they should not, such as deleting topics or joining the wrong consumer group.

Principals

All ACLs require a principal. A principal is composed of two parts: the type, and the name. Redpanda supports the type "User". When you create user "bar", Redpanda expects you to add ACLs for "User:bar".

The --allow-principal and --deny-principal flags add this prefix for you, if necessary.

The special character * matches any name, meaning an ACL with principal User:* grants or denies the permission for any user.

To set multiple principals in a single comma-separated string, you must enclose the string with quotes. Otherwise, rpk splits the string on commas and fails to read the option correctly.

For example, use double quotes:

rpk acl create --allow-principal="\"C=UK,ST=London,L=London,O=Redpanda,OU=engineering,CN=__schema_registry\""

Alternatively, use single quotes:

rpk acl create --allow-principal='"C=UK,ST=London,L=London,O=Redpanda,OU=engineering,CN=__schema_registry"'

Hosts

Hosts can be seen as an extension of the principal and can effectively gate where the principal can connect from. When creating ACLs, unless otherwise specified, the default host is the wildcard *, which allows or denies the principal from all hosts.

When specifying hosts, you must pair the --allow-host flag with the --allow-principal flag and the --deny-host flag with the --deny-principal flag.

Resources

A resource is what an ACL allows or denies access to. There are four resources within Redpanda: topics, groups, cluster, and transactionalid. Names for each of these resources can be specified with their respective flags.

Resources combine with the operation that is allowed or denied on that resource. By default, resources are specified on an exact name match (a "literal" match).

Use the --resource-pattern-type flag to specify that a resource name is "prefixed", meaning to allow anything with the given prefix. A literal name of "foo" matches only the topic "foo", while the prefixed name of "foo-" matches both "foo-bar" and "foo-jazz". The special wildcard resource name '' matches any name of the given resource type (--topic '' matches all topics).

Operations

Paired with resources, operations are the actions that are allowed or denied. Redpanda supports the following operations:

Operation Description

ALL

Allows all operations below.

READ

Allows reading a given resource.

WRITE

Allows writing to a given resource.

CREATE

Allows creating a given resource.

DELETE

Allows deleting a given resource.

ALTER

Allows altering non-configurations.

DESCRIBE

Allows querying non-configurations.

DESCRIBE_CONFIGS

Allows describing configurations.

ALTER_CONFIGS

Allows altering configurations.

Producing/Consuming

The following operations are necessary for each individual client request, where resource corresponds to the resource flag, and "for xyz" corresponds to the resource names in the request:

PRODUCING/CONSUMING

    Produce      WRITE on TOPIC for topics
                 WRITE on TRANSACTIONAL_ID for the transaction.id

    Fetch        READ on TOPIC for topics

    ListOffsets  DESCRIBE on TOPIC for topics

    Metadata     DESCRIBE on TOPIC for topics
                 CREATE on CLUSTER for kafka-cluster (if automatically creating topics)
                 CREATE on TOPIC for topics (if automatically creating topics)

    OffsetForLeaderEpoch  DESCRIBE on TOPIC for topics

GROUP CONSUMING

    FindCoordinator  DESCRIBE on GROUP for group
                     DESCRIBE on TRANSACTIONAL_ID for transactional.id (transactions)

    OffsetCommit     READ on GROUP for groups
                     READ on TOPIC for topics

    OffsetFetch      DESCRIBE on GROUP for groups
                     DESCRIBE on TOPIC for topics

    OffsetDelete     DELETE on GROUP for groups
                     READ on TOPIC for topics

    JoinGroup        READ on GROUP for group
    Heartbeat        READ on GROUP for group
    LeaveGroup       READ on GROUP for group
    SyncGroup        READ on GROUP for group

TRANSACTIONS (including FindCoordinator above)

    AddPartitionsToTxn  WRITE on TRANSACTIONAL_ID for transactional.id
                        WRITE on TOPIC for topics

    AddOffsetsToTxn     WRITE on TRANSACTIONAL_ID for transactional.id
                        READ on GROUP for group

    EndTxn              WRITE on TRANSACTIONAL_ID for transactional.id

    TxnOffsetCommit     WRITE on TRANSACTIONAL_ID for transactional.id
                        READ on GROUP for group
                        READ on TOPIC for topics

ADMIN

    CreateTopics      CREATE on CLUSTER for kafka-cluster
                      CREATE on TOPIC for topics
                      DESCRIBE_CONFIGS on TOPIC for topics,
                      for returning topic configs on create

    CreatePartitions  ALTER on TOPIC for topics

    DeleteTopics      DELETE on TOPIC for topics
                      DESCRIBE on TOPIC for topics, if deleting by topic ID
                      (in addition to prior ACL)

    DeleteRecords     DELETE on TOPIC for topics

    DescribeGroup     DESCRIBE on GROUP for groups

    ListGroups        DESCRIBE on GROUP for groups
                      or, DESCRIBE on CLUSTER for kafka-cluster

    DeleteGroups      DELETE on GROUP for groups

    DescribeConfigs   DESCRIBE_CONFIGS on CLUSTER for cluster (broker describing)
                      DESCRIBE_CONFIGS on TOPIC for topics (topic describing)

    AlterConfigs      ALTER_CONFIGS on CLUSTER for cluster (broker altering)
                      ALTER_CONFIGS on TOPIC for topics (topic altering)

To get this information at the CLI, run:

rpk acl --help-operations

In flag form to set up a general producing/consuming client, you can invoke rpk acl create up to three times with the following (including your --allow-principal):

--operation write,read,describe --topic [topics]
--operation describe,read --group [group.id]
--operation describe,write --transactional-id [transactional.id]

Permissions

A client can be allowed access or denied access. By default, all permissions are denied. You only need to specifically deny a permission if you allow a wide set of permissions and then want to deny a specific permission in that set. You could allow all operations, and then specifically deny writing to topics.

Management

Commands for managing users and ACLs work on a specific ACL basis, but listing and deleting ACLs works on filters. Filters allow matching many ACLs to be printed, listed, and deleted at the same time. Because this can be risky for deleting, the delete command prompts for confirmation by default.

rpk for managing users and ACLs

The rpk acl command manages your ACLs as well as your SASL/SCRAM users. (If you’re on Kubernetes, you can use kubectl exec to run rpk commands.)

rpk acl [command] [flags]

For example, to create a user:

rpk acl user create Jack \
--password '<password>' \
-X admin.hosts=localhost:9644

Created user 'Jack'

Here are all the available commands and how they interact with Redpanda:

Command Protocol Default Port

user

Admin API

9644

list

Kafka API

9092

create

Kafka API

9092

delete

Kafka API

9092

To get more information, see rpk acl.

Global flags

Every rpk acl command can use these flags:

Flag Description

--admin-api-tls-cert

The certificate to be used for TLS authentication with the Admin API.

--admin-api-tls-enabled

Enable TLS for the Admin API (not necessary if specifying custom certificates). This is assumed as true when passing other --admin-api-tls flags.

--admin-api-tls-key

The certificate key to be used for TLS authentication with the Admin API.

--admin-api-tls-truststore

The truststore to be used for TLS communication with the Admin API.

-X brokers

Comma-separated list of broker ip:port pairs (for example, --brokers '192.168.78.34:9092,192.168.78.35:9092,192.179.23.54:9092' ). Alternatively, you can set the RPK_BROKERS environment variable with the comma-separated list of broker addresses.

--config

Redpanda configuration file. If not set, the file is searched in the default locations.

-h, --help

Help.

--password

SASL password to be used for authentication.

--sasl-mechanism

The authentication mechanism to use. Supported values: SCRAM-SHA-256, SCRAM-SHA-512.

--tls-cert

The certificate to be used for TLS authentication with the broker.

--tls-enabled

Enable TLS for the Kafka API (not necessary if specifying custom certificates). This is assumed to be true when passing other --tls flags.

--tls-key

The certificate key to be used for TLS authentication with the broker.

--tls-truststore

The truststore to be used for TLS communication with the broker.

--user

SASL user to be used for authentication.

Create ACLs

With the create command, every ACL combination is a created ACL. At least one principal, one host, one resource, and one operation are required to create a single ACL.

rpk acl create/delete [globalACLFlags] [localFlags]

You can use the global flags and some other local flags. Following are the available local flags:

Flag Description

--allow-host

Host for which access will be granted (repeatable).

--allow-principal

Principals to which permissions will be granted (repeatable).

--cluster

Whether to grant ACLs to the cluster.

--deny-host

Host from which access will be denied (repeatable).

--deny-principal

Principal to which permissions will be denied (repeatable).

--group

Group to grant ACLs for (repeatable).

-h, --help

Help.

--name-pattern

The name pattern type to be used when matching the resource names.

--operation

Operation that the principal will be allowed or denied. Can be passed many times.

--resource-pattern-type

Pattern to use when matching resource names (literal or prefixed) (default "literal").

--topic

Topic to grant ACLs for (repeatable).

--transactional-id

Transactional IDs to grant ACLs for (repeatable).

Examples:

To allow all permissions to user bar on topic "foo" and group "g", run:

rpk acl create --allow-principal bar --operation all --topic foo --group g

To allow read permissions to all users on topics biz and baz, run:

rpk acl create --allow-principal '*' --operation read --topic biz,baz

To allow write permissions to user buzz to transactional id "txn", run:

rpk acl create --allow-principal User:buzz --operation write --transactional-id txn

List and delete ACLs

List and delete for ACLs have a multiplying effect (similar to create ACL), but delete is more advanced. List and delete work on a filter basis. Any unspecified flag defaults to matching everything (all operations, or all allowed principals, and so on).

To ensure that you don’t accidentally delete more than you intend, this command prints everything that matches your input filters and prompts for a confirmation before the delete request is issued. Anything matching more than 10 ACLs also asks for confirmation.

If no resources are specified, all resources are matched. If no operations are specified, all operations are matched.

You can opt in to matching everything. For example, --operation any matches any operation.

The --resource-pattern-type, defaulting to any, configures how to filter resource names:

  • any returns exact name matches of either prefixed or literal pattern type

  • match returns wildcard matches, prefix patterns that match your input, and literal matches

  • prefix returns prefix patterns that match your input (prefix "fo" matches "foo")

  • literal returns exact name matches

To list or delete ACLs, run:

rpk acl list/delete [globalACLFlags] [localFlags]

You can use the global flags and some other local flags. Following are the available local flags:

Flag Description

--allow-host

Allowed host ACLs to list/remove. (repeatable)

--allow-principal

Allowed principal ACLs to list/remove. (repeatable)

--cluster

Whether to list/remove ACLs to the cluster.

--deny-host

Denied host ACLs to list/remove. (repeatable)

--deny-principal

Denied principal ACLs to list/remove. (repeatable)

-d, --dry

Dry run: validate what would be deleted.

--group

Group to list/remove ACLs for. (repeatable)

-h, --help

Help.

--no-confirm

Disable confirmation prompt.

--operation

Operation to list/remove. (repeatable)

-f, --print-filters

Print the filters that were requested. (failed filters are always printed)

--resource-pattern-type

Pattern to use when matching resource names. (any, match, literal, or prefixed) (default "any")

--topic

Topic to list/remove ACLs for. (repeatable)

--transactional-id

Transactional IDs to list/remove ACLs for. (repeatable)

User

This command manages the SCRAM users. If SASL is enabled, a SCRAM user talks to Redpanda, and ACLs control what your user has access to. Using SASL requires setting kafka_enable_authorization: true in the Redpanda section of your redpanda.yaml.

rpk acl user [command] [globalACLFlags] [globalUserFlags]

Following are the available global user flags:

Flag Description Supported Value

-X admin.hosts

The comma-separated list of Admin API addresses (IP:port). You must specify one for each broker.

strings

-h, --help

-h, --help

Help.

User create

This command creates a single SASL/SCRAM user with the given password, and optionally with a custom mechanism. The mechanism determines which authentication flow the client uses for this user/password. Redpanda rpk supports the following mechanisms: SCRAM-SHA-256 (default) and SCRAM-SHA-512, which is the same flow but uses sha512. To use GSSAPI, see Enable Kerberos.

Before a created SASL account can be used, you must also create ACLs to grant the account access to certain resources in your cluster.

To create a SASL/SCRAM user, run:

rpk acl user create [user] -p [password] [globalACLFlags] [globalUserFlags] [localFlags]

Here are the local flags:

Flag Description

-h, --help

Help.

--mechanism

SASL mechanism to use: scram-sha-256 or scram-sha-512. Default is scram-sha-256.

User delete

This command deletes the specified SASL account from Redpanda. This does not delete any ACLs that may exist for this user. You may want to re-create the user later, as well, not all ACLs have users that they describe (instead they are for wildcard users).

rpk acl user delete [USER] [globalACLFlags] [globalUserFlags]

User list

This command lists SASL users.

rpk acl user list [globalACLFlags] [globalUserFlags]

You can also use the shortened version changing list to ls.

Suggested reading

Suggested labs

Search all labs