Labs

Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console

This lab walks you through integrating Azure Entra ID (formerly Azure AD) with Redpanda and Redpanda Console for unified identity using OpenID Connect (OIDC). This integration allows you to use Azure Entra ID for authentication in Redpanda and Redpanda Console.

Redpanda does not support mapping Entra ID groups or other token claims to Redpanda roles. As a result, authorization decisions (who can do what) are not made by Entra ID but must be configured directly in Redpanda using ACLs or role bindings. Azure Entra ID authenticates the user, but Redpanda enforces access.

Prerequisites

This lab is intended for Linux and macOS users. If you are using Windows, you must use the Windows Subsystem for Linux (WSL) to run the commands in this lab.

Set up Azure Entra ID

This section guides you through creating an Azure Entra ID application registration and configuring it for use with Redpanda. You’ll set up the necessary permissions, scopes, and claims to ensure proper authentication and authorization.

Create an application registration

In this section, you’ll register a new application in Entra ID that Redpanda will use to authenticate.

  1. Sign in to Azure Portal.

  2. Navigate to Azure Active Directory > App registrations > New registration.

  3. Configure:

  4. Click Register.

See also: Register an application.

Set access token version to v2

Redpanda and Redpanda Console require that your identity provider (IdP) issues JWT-encoded access tokens that can be cryptographically verified. To ensure this, the IdP must support OpenID Connect (OIDC), not just OAuth 2.0.

By default, OAuth 2.0 allows for opaque tokens, which Redpanda does not support. OIDC extends OAuth 2.0 by specifying how tokens should be structured, signed, and validated using a discovery document and public keys.

In Azure Entra ID, you must configure the application to issue v2.0 JWT access tokens:

  1. In the App Registration page, go to Manage > Manifest.

  2. Set accessTokenAcceptedVersion to 2:

    "accessTokenAcceptedVersion": 2

  3. Click Save.

Azure Entra ID does not include claims like email in access tokens by default. You can add them by going to Token Configuration > + Add optional claim, then selecting email for Access token type.

Define a custom scope

Scopes are required to explicitly request JWT access tokens.

  1. Go to Expose an API > Set Application ID URI (accept the default if prompted).

  2. Click Add a scope and enter:

    • Scope name: entraid.v2-access-tokens

    • Who can consent: Admins and users

    • Display name: Entra ID v2 access tokens

    • Description: Allows Redpanda to request v2 access tokens

  3. Click Add scope.

See also: Expose an API.

Create a client secret

This secret is used by Redpanda Console to authenticate with Entra ID.

  1. Go to Certificates & secrets > New client secret.

  2. Add a description and choose an expiration (12-24 months recommended).

  3. Save the generated secret securely.

Gather configuration values

Collect these values to configure Redpanda and Redpanda Console later.

Key Location Example

Client ID

App Registration Overview

ecc74380-7c64-4283-9fa1-03a37b9054b7

Tenant ID

App Registration Overview

9a95fd9e-005d-487a-9a01-d08c1eab2757

Client Secret

Certificates & Secrets

GENERATED_SECRET_VALUE

Issuer URL

Manual: use Tenant ID

https://login.microsoftonline.com/<tenant-id>/v2.0

Verify the token configuration

This step confirms you receive a token with expected claims.

  1. Run a test OAuth client credentials flow:

    curl -X POST "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token" \
  -d "client_id=<client-id>" \
  -d "scope=api://<client-id>/.default" \
  -d "client_secret=<client-secret>" \
  -d "grant_type=client_credentials"

  2. Decode the returned token at jwt.io and verify:

    • iss: matches the issuer URL

    • aud: matches your client ID

    • ver: is 2.0

    • sub: is present

  3. Save the sub claim value for later use.

Application registration final checklist

  • accessTokenAcceptedVersion set to 2

  • Custom scope entraid.v2-access-tokens created

Run the lab

  1. Clone this repository:

    git clone https://github.com/redpanda-data/redpanda-labs.git

  2. Change into the docker-compose/oidc/ directory:

    cd redpanda-labs/docker-compose/oidc

  3. Set the REDPANDA_VERSION environment variable to a supported version of Redpanda. For all available versions, see the GitHub releases.

    For example:

    export REDPANDA_VERSION=v26.1.9

  4. Set the REDPANDA_CONSOLE_VERSION environment variable to the version of Redpanda Console that you want to run. For all available versions, see the GitHub releases.

    You must use Redpanda Console version v3.1.0 or later to deploy this lab.

    For example:

    export REDPANDA_CONSOLE_VERSION=v3.7.3

  5. Create a .env file and add the following environment variables:

    .env
    TENANT_ID=<tenant-id>
OIDC_CLIENT_ID=<client-id>
OIDC_CLIENT_SECRET=<client-secret>
JWT_SIGNING_KEY=<jwt-signing-key>

    Redpanda Console expects configuration to be passed as environment variables, where each YAML key is converted to uppercase and underscores are added for each level of nesting. For example, authentication.oidc.clientSecret becomes AUTHENTICATION_OIDC_CLIENTSECRET.

    To simplify this, user-friendly environment variables, such as OIDC_CLIENT_SECRET, are set here and then mapped to the required format in the docker-compose.yml file.

    This pattern helps separate sensitive or dynamic values from implementation-specific keys. See Configure Redpanda Console for details.

  6. Start Redpanda and Redpanda Console in Docker by running the following command:

    docker compose up --detach --wait

  7. Set up your rpk profile on your local machine:

    rpk profile create oidc-lab --from-profile profile.yaml

  8. Update the Redpanda cluster configuration.

    Redpanda does not support environment variables for cluster-level properties. To configure OIDC authentication, use the following rpk CLI commands:

    # Define at least one superuser. This is required for managing ACLs and cluster config.
# Add your OIDC user’s `sub` claim here to grant superuser privileges.
rpk cluster config set superusers '["superuser", "<oidc-sub>"]'

# Enable SASL-based authentication on the Kafka API.
rpk cluster config set enable_sasl true

# Require authentication on the Admin API. Without this, the Admin API remains open.
rpk cluster config set admin_api_require_auth true

# Enable supported SASL mechanisms for the Kafka API.
# SCRAM is for password-based users. OAUTHBEARER is for OIDC tokens.
rpk cluster config set sasl_mechanisms '["SCRAM", "OAUTHBEARER"]'

# Allow both BASIC (SCRAM) and OIDC authentication for HTTP APIs.
rpk cluster config set http_authentication '["BASIC", "OIDC"]'

# Set the expected audience (aud claim) for validating OIDC tokens.
rpk cluster config set oidc_token_audience "<client-id>"

# Configure the OIDC discovery URL (typically ends in /.well-known/openid-configuration).
rpk cluster config set oidc_discovery_url "https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration"

# Map the OIDC user identity from the token’s `sub` claim (default).
rpk cluster config set oidc_principal_mapping "$.sub"

# Allow for some clock drift between Redpanda and the IdP.
rpk cluster config set oidc_clock_skew_tolerance 600

# Optional: Enable automatic topic creation on first access.
rpk cluster config set auto_create_topics_enabled true

    Replace all placeholder values with the values you gathered from the Azure Entra ID app registration:

    • <tenant-id>: Tenant ID

    • <client-id>: Client ID

    • <oidc-user>: Sub value from the token

  9. Enable authentication on HTTP Proxy and Schema Registry.

    To ensure secure access to the HTTP-based APIs (HTTP Proxy and Schema Registry), you must enable authentication on those listeners by setting authentication_method: http_basic.

    1. Open a terminal and run:

      docker exec -it redpanda-0 bash

    2. Open the redpanda.yaml config file:

      nano /etc/redpanda/redpanda.yaml

    3. Add authentication_method: http_basic to the HTTP Proxy (pandaproxy) and Schema Registry listeners to enable authentication for those API endpoints. For example:

      pandaproxy:
    pandaproxy_api:
        - address: 0.0.0.0
          port: 8082
          name: internal
          authentication_method: http_basic
        - address: 0.0.0.0
          port: 18082
          name: external
          authentication_method: http_basic
    advertised_pandaproxy_api:
        - address: redpanda-0
          port: 8082
          name: internal
          authentication_method: http_basic
        - address: localhost
          port: 18082
          name: external
          authentication_method: http_basic
schema_registry:
    schema_registry_api:
        - address: 0.0.0.0
          port: 8081
          name: internal
          authentication_method: http_basic
        - address: 0.0.0.0
          port: 18081
          name: external
          authentication_method: http_basic

    4. Save the file and exit the editor.

    5. Exit the container:

      exit

    6. Restart the Redpanda broker:

      docker restart redpanda-0

    7. Repeat this process for each broker (redpanda-1 and redpanda-2).

  10. Open Redpanda Console in your browser at http://localhost:8080/login.

  11. Click Log in with OIDC and enter the login details of your OIDC user.

    You should be redirected to the Redpanda Console dashboard.

You are now logged in to Redpanda Console using OIDC authentication with Azure Entra ID. You can now use Redpanda Console to manage your Redpanda cluster and monitor its performance.

Connect a Kafka client with OIDC

You can test OIDC client authentication by connecting a Kafka client, such as KafkaJS, to Redpanda.

KafkaJS supports OIDC through the oauthBearerProvider configuration. This function must return a valid access token that the client uses to authenticate with Redpanda through the SASL/OAUTHBEARER mechanism.

The provided client.js script demonstrates this by:

  • Authenticating to Azure Entra ID using the client credentials flow.

  • Retrieving a JWT access token.

  • Using that token to produce a message to Redpanda.

When using the client credentials flow, Azure Entra ID issues tokens for applications, not users. These tokens do not include user-specific claims like email or preferred_username. Instead, the token includes the sub claim (subject), which uniquely identifies the application.

  1. Find the sub value by running the client script:

    npm install && node client.js get-sub-value

    Look for the sub field, which will look like a UUID:

    "sub": "ae775e64-5853-42cb-b62a-e092c7c5288b"

  2. Use rpk to assign a role to the application’s sub value:

    rpk security acl create \
  --allow-principal User:<sub-claim> \
  --operation write,read,describe,create \
  --topic test-topic

  3. Run the client script again to start producing messages to the test topic:

    node client.js

    Output:

    [Kafka] Connecting producer...
[OIDC] Fetching new access token...
[Kafka] Sending message...
[Kafka] Message sent successfully.
[Kafka] Producer disconnected.

    You should now see the message produced to the test topic in your Redpanda cluster.

  4. Consume the message from the test topic:

    rpk topic consume test-topic --num 1

    Example output:

    {
  "topic": "test-topic",
  "value": "Hello from OIDC client!",
  "timestamp": 1746623458222,
  "partition": 0,
  "offset": 0
}

Token refresh

The client.js script implements automatic token refreshing using simple-oauth2. When the token is about to expire, it fetches a new one in the background, keeping the Kafka client authenticated seamlessly.

For long-running apps, this pattern avoids token expiry errors and ensures smooth reconnections.

Clean up

To shut down and delete the containers along with all your cluster data:

docker compose down -v

Troubleshoot OIDC login

If you encounter issues logging in to Redpanda Console, check the following in your Redpanda Console configuration:

  • Ensure the issuerUrl matches the issuer URL from the application registration.

  • Verify the clientId and clientSecret match those from the application registration.

  • Check the redirectUrl matches the redirect URI set in the application registration.

  • Ensure the additionalScopes includes the custom scope you created in the application registration.

  • Verify the oidc_principal_mapping matches the claim you want to use for user mapping.

  • Check the oidc_token_audience in your Redpanda configuration matches the client ID from the application registration.

OIDC limitations

  • Redpanda requires JWT-formatted access tokens (not ID tokens) for Kafka API authentication using SASL/OAUTHBEARER. Access tokens issued by some IdPs, such as Google, are opaque and not supported.

  • The rpk CLI does not support OIDC login.

  • Redpanda requires OIDC principals to be set as superusers to access the Admin API. Granular authorization is not supported.

  • The rpk CLI does not support the SASL/OAUTHBEARER mechanism for deploying data transforms. Use SASL/SCRAM instead.

Suggested reading