Authentication
Redpanda Cloud ensures the highest level of authentication for both users and services.
User authentication
Redpanda provides user authentication to your Redpanda organization through email or single sign-on.
Redpanda Cloud can authenticate users with emails and passwords. Passwords are hashed (a one-way function that makes the original value unrecoverable, and effectively encrypted) and salted at rest using bcrypt.
Single sign-on
Redpanda Cloud can authenticate users with single sign-on (SSO) to an OIDC-based identity provider (IdP). Redpanda integrates with any OIDC-compliant IdP that supports discovery, including Okta, Microsoft Entra ID, Auth0, Active Directory Federation Services (AD-FS), and JumpCloud. After SSO is enabled for an organization, new users in that organization can authenticate with SSO.
You must integrate your IdP with Redpanda Cloud to use SSO. On the Users page, users with admin permission see a Single sign-on tab and can add connections for up to two different IdPs. Enter the client ID, client secret, and discovery URI for the IdP. (See your IdP documentation for these values. The discovery URI may be called something different, like the well known URL or the issuer_url
.)
By default, the connection is added in a disabled state. Edit the connection to enable it. You can choose to enable auto-enroll in the connection, which provides new users signing in from that IdP access to your Redpanda organization. When you enable auto-enroll, you select to assign a read, write, or admin role to users who log in with that IdP. Set up is different for most IdPs.
Deleting an SSO connection also deletes all users attached to it. |
Integrate with Okta
To integrate with Okta, follow the Okta documentation to create an application within Okta for Redpanda. The Redpanda callback location (that is, the redirect location where Okta sends the user) is the following:
https://auth.prd.cloud.redpanda.com/login/callback
Okta provides the following fields required for SSO configuration on the Redpanda Users page: clientId
, clientSecret
, and discoveryUrl
. The discovery URL for Okta generally looks like the following (where an_id
could be “default”):
https://<orgname>.okta.com/oauth2/<an_id>/.well-known/openid-configuration
Integrate with Microsoft Entra ID
To integrate with Azure Entra ID, follow the Microsoft documentation to create an OIDC enterprise (web) application:
-
In the Entra App Gallery, on the Create your own application page:
-
Select Register an application to integrate with Microsoft Entra ID.
-
For the name of your app, enter
Redpanda Cloud
. -
Click Create.
-
-
On the Register an application page:
-
For Supported account types, select Accounts in this organizational directory only (Redpanda only - Single tenant).
-
For Redirect URI, select Web platform with the Callback URL found in Redpanda Cloud. In Redpanda Cloud, navigate to Users: Single sign-on, and click Add connection. Copy the Callback URL, and paste it into the Azure Redirect URI field.
-
Click Register.
-
-
On the Azure Enterprise applications page, you can now search for the Redpanda Cloud app to assign users access to Redpanda Cloud.
-
On the Azure app for Redpanda Cloud, click Endpoints, and copy the OpenID Connect metadata document URL endpoint.
-
In Redpanda Cloud, on Users: Single sign-on, paste that endpoint address into the Discovery URI field. Then, complete the SSO configuration:
-
For Client ID, copy and paste the Application (client) ID from the Azure app for Redpanda Cloud.
-
For Client secret, copy and paste the secret you get from adding a client secret on the Certificates & secrets page for the Azure app for Redpanda Cloud.
-
For Realm, enter your Azure Entra ID tenant domain name.
-
Click Save.
-
On the Redpanda Cloud SSO page, edit your new Entra ID connection to enable SSO.
Users with an email address with that realm (domain) can now access your Redpanda Cloud account.
You can continue to configure your Azure Enterprise app page for Redpanda Cloud, but there is no need to configure JWT-based claims or API permissions.
-
Service authentication
Each Redpanda Cloud data plane runs its own dedicated agent, which authenticates and connects against the control plane over a single TLS 1.2 encrypted TCP connection.
Different Redpanda APIs support different authentication methods. For GCP, you can simultaneously enable mTLS and SASL for Kafka API, and mTLS and Basic authentication for the HTTP APIs (HTTP Proxy and Schema Registry). If you enable both mTLS and SASL on GCP clusters, Redpanda creates two distinct listeners: an mTLS listener operating on one port and a SASL listener operating on a different port.
Cloud provider | API | Supported authentication methods |
---|---|---|
AWS |
Kafka API |
|
HTTP Proxy |
|
|
Schema Registry |
|
|
GCP See Enable mTLS and SASL. |
Kafka API |
|
HTTP Proxy |
|
|
Schema Registry |
|
|
Azure |
Kafka API |
|
HTTP Proxy |
|
|
Schema Registry |
|
-
Kafka API: Redpanda Cloud enables SASL/SCRAM authentication over TLS 1.2 as well as mTLS to authenticate Kafka clients connecting to Redpanda clusters over the TCP endpoint or listener.
-
HTTP Proxy and Schema Registry: Authentication is done through an HTTP Basic Authentication header encrypted over TLS 1.2.
The following features use IAM policies to generate dynamic and short-lived credentials to interact with cloud provider APIs:
-
Data plane agent
-
Tiered Storage
-
Redpanda Console
-
Managed connectors
IAM policies have constrained permissions so that each service can only access or manage its own data plane-scoped resources, following the principle of least privilege.
Configure service authentication
Redpanda Cloud supports mTLS or SASL authentication for Kafka API, HTTP Proxy, and Schema Registry.
When you create a new cluster using the Cloud UI, the cluster is enabled by default with SASL for Kafka API, and Basic authentication for HTTP Proxy and Schema Registry.
Requirements
mTLS authentication is supported on AWS and GCP clusters only. |
If you want to enable mTLS authentication:
-
You must use the Cloud API to create a new mTLS-enabled cluster.
-
You must also use the Cloud API to update an existing cluster to switch to mTLS authentication for Kafka API.
-
You can use the Cloud UI to update an existing cluster to switch to mTLS authentication for HTTP Proxy and Schema Registry only.
-
To enable mTLS and SASL (or Basic authentication) simultaneously on GCP clusters, you must use the Cloud API to create a new cluster or update an existing cluster.
To configure service authentication in your cluster using the Cloud API, you must have the following:
-
A service account in your Redpanda organization. You must have administrative privileges to create a service account.
-
Access to the Cloud API to enable mTLS or both SASL and mTLS. You use the service account to obtain an access token. You then make a request to the Control Plane API, passing the access token in the authorization header.
Authenticate to the Cloud API
-
Create a service account in your organization, if you haven’t already. Go to the Service account tab of the Organization IAM page in the Redpanda Cloud UI and click Create service account to create a service account. Enter a name and description.
-
Retrieve the client ID and secret by clicking Copy ID and Copy Secret.
-
Obtain an access token by making a
POST
request tohttps://auth.prd.cloud.redpanda.com/oauth/token
with the ID and secret in the request body.AUTH_TOKEN=`curl -s --request POST \ --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \ --header 'content-type: application/x-www-form-urlencoded' \ --data grant_type=client_credentials \ --data client_id=<client-id> \ --data client_secret=<client-secret> \ --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`
bashMake sure to replace the following variables:
Placeholder variable Description <client-id>
Client ID.
<client-secret>
Client secret.
Enable mTLS authentication
For clusters with mTLS authentication, Redpanda creates a dedicated mTLS-enabled listener for each API service (Kafka API, HTTP Proxy, or Schema Registry) where you’ve enabled this authentication method. After you enable mTLS, get the API endpoints and verify that mTLS authentication is in effect.
If you enable mTLS authentication, you cannot disable it later. |
Create a new cluster with mTLS enabled
-
Follow the steps to to create a resource group and network for BYOC or Dedicated, if you haven’t already. You’ll need the resource group ID and network ID to create a cluster in the next step.
-
Make a
POST /v1/clusters/{cluster.id}
request to create a new cluster with mTLS enabled.
The following example enables mTLS for Kafka API. To enable mTLS for HTTP Proxy and Schema Registry, add the http_proxy.mtls and schema_registry.mtls fields to the request body. You can choose to enable mTLS for any combination of the three services.
|
Show example request to enable mTLS for Kafka API
CLUSTER_CREATE_BODY=`cat << EOF
{
"cluster": {
"cloud_provider": "CLOUD_PROVIDER_GCP",
"connection_type": "CONNECTION_TYPE_PRIVATE",
"name": "<cluster-name>",
"resource_group_id": "<resource-group-id>",
"network_id": "<network-id>",
"region": "<region>",
"zones": [ <zones> ],
"throughput_tier": "<tier>",
"type": "<cluster-type>",
"kafka_api": {
"mtls": {
"enabled": true,
"ca_certificates_pem": ["<ca-certificate-pem>"],
"principal_mapping_rules": ["<principal-mapping-rule>"]
}
}
}
}
EOF`
curl -v -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_CREATE_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`
Make sure to replace the following variables:
Placeholder variable | Description |
---|---|
|
ID of the Redpanda cluster. |
|
Name of the Redpanda cluster. |
|
ID of the resource group. |
|
ID of the network. |
|
The region where the cluster is created. For example, |
|
The zones where the cluster is created. For example, |
|
The usage tier of the cluster. For example, . |
|
The Redpanda cluster type, |
|
A trusted Kafka client CA certificate in PEM format. The |
|
Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals. For example, the mapping rule
See Configure Authentication for more details on principal mapping rules. The |
The Create Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET
request to the following endpoint:
curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>
When the operation state is COMPLETED
, you can verify that mTLS is enabled for the API endpoints.
Update an existing cluster to use mTLS
Make a PATCH /v1/clusters/{cluster.id}
request to enable mTLS for the Kafka API on a cluster.
The following code block shows a request to enable mTLS for Kafka API. To enable mTLS for HTTP Proxy and Schema Registry, add the http_proxy.mtls
and schema_registry.mtls
fields to the request body:
Show example request
CLUSTER_PATCH_BODY=`cat << EOF
{
"kafka_api": {
"mtls": {
"enabled": true,
"ca_certificates_pem": ["<ca-certificate-pem>"],
"principal_mapping_rules": ["<principal-mapping-rule>"]
}
}
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`
Make sure to replace the following variables:
Placeholder variable | Description |
---|---|
|
ID of Redpanda cluster. |
|
A trusted Kafka client CA certificate in PEM format. The |
|
Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals. For example, the mapping rule
See Configure Authentication for more details on principal mapping rules. The |
The Update Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET
request to the following endpoint:
curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>
When the operation state is COMPLETED
, you can verify that mTLS is enabled for the API endpoints.
Enable mTLS and SASL
Enabling mTLS and SASL simultaneously is available for GCP clusters only. To unlock this feature for your account, contact your Customer Success Manager. |
You can choose to enable mTLS and SASL simultaneously for the Kafka API, and mTLS and Basic authentication for HTTP Proxy and Schema Registry. The sasl
field in the API request examples toggle both SASL and Basic authentication.
Create a new cluster with both mTLS and SASL enabled
-
Follow the steps to to create a resource group and network for BYOC or Dedicated, if you haven’t already done so. You’ll need the resource group ID and network ID to create a cluster in the next step.
-
Make a
POST /v1/clusters/{cluster.id}
request to create a new cluster with both mTLS and SASL or Basic authentication enabled.You can enable mTLS and SASL or Basic authentication for any combination of the three services. For example, if you want to enable mTLS and SASL simultaneously for Kafka API and mTLS and Basic authentication simultaneously for Schema Registry only, leave out the entire
http_proxy
block from the request body. If you want to enable mTLS only for Kafka API, and mTLS and Basic authentication for HTTP Proxy and Schema Registry, leave out thekafka_api.sasl
field.Show example request
CLUSTER_CREATE_BODY=`cat << EOF { "cluster": { "cloud_provider": "CLOUD_PROVIDER_GCP", "connection_type": "CONNECTION_TYPE_PRIVATE", "name": "<cluster-name>", "resource_group_id": "<resource-group-id>", "network_id": "<network-id>", "region": "<region>", "zones": [ <zones> ], "throughput_tier": "<tier>", "type": "<cluster-type>", "kafka_api": { "mtls": { "enabled": true, "ca_certificates_pem": ["<ca-certificate-pem>"], "principal_mapping_rules": ["<principal-mapping-rule>"] }, "sasl": { "enabled": true } }, "http_proxy": { "mtls": { "enabled": true, "ca_certificates_pem": ["<ca-certificate-pem>"] }, "sasl": { "enabled": true } }, "schema_registry": { "mtls": { "enabled": true, "ca_certificates_pem": ["<ca-certificate-pem>"] }, "sasl": { "enabled": true } } } } EOF` curl -v -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$CLUSTER_CREATE_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`
bashMake sure to replace the following variables:
Placeholder variable Description <cluster-id>
ID of Redpanda cluster.
<cluster-name>
Name of the Redpanda cluster.
<resource-group-id>
ID of the resource group.
<network-id>
ID of the network.
<region>
The region where the cluster is created. For example,
us-central1
.<zones>
The zones where the cluster is created. For example,
["us-central1-a", "us-central1-b", "us-central1-c"]
.<tier>
The usage tier of the cluster. For example, .
<cluster-type>
The Redpanda cluster type,
TYPE_BYOC
orTYPE_DEDICATED
.<ca-certificate-pem>
A trusted Kafka client CA certificate in PEM format. The
ca_certificates_pem
field accepts a list of certificates.<principal-mapping-rule>
Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals.
For example, the mapping rule
RULE:.*CN=([^,]+).*/\\$1/
maps the following certificate subject to a principal namedtest
:Subject: C=US, ST=IL, L=Chicago, O=redpanda, OU=cloud, CN=test, emailAddress=test123@redpanda.com
See Configure Authentication for more details on principal mapping rules. The
principal_mapping_rules
field accepts a list of rules.
The Create Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET
request to the following endpoint:
curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>
When the operation state is COMPLETED
, you can verify that mTLS is enabled for the API endpoints.
Update an existing cluster to use mTLS and SASL
Make a PATCH /v1/clusters/{cluster.id}
request to enable mTLS and SASL on an existing cluster.
You can choose to enable mTLS and SASL or Basic authentication for any combination of the three services. For example, if you want to enable mTLS and SASL simultaneously for Kafka API and mTLS and Basic authentication simultaneously for Schema Registry only, leave out the entire http_proxy
block from the request body. If you want to enable mTLS only for Kafka API, and mTLS and Basic authentication for HTTP Proxy and Schema Registry, leave out the kafka_api.sasl
field.
Show example request
CLUSTER_PATCH_BODY=`cat << EOF
{
"kafka_api": {
"mtls": {
"enabled": true,
"ca_certificates_pem": ["<ca-certificate-pem>"],
"principal_mapping_rules": ["<principal-mapping-rule>"]
},
"sasl": {
"enabled": true
}
},
"schema_registry": {
"mtls": {
"enabled": true,
"ca_certificates_pem": ["<ca-certificate-pem>"]
},
"sasl": {
"enabled": true
}
},
"http_proxy": {
"mtls": {
"enabled": true,
"ca_certificates_pem": ["<ca-certificate-pem>"]
},
"sasl": {
"enabled": true
}
}
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`
Make sure to replace the following variables:
Placeholder variable | Description |
---|---|
|
ID of Redpanda cluster. |
|
A trusted Kafka client CA certificate in PEM format. The |
|
Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals. For example, the mapping rule
See Configure Authentication for more details on principal mapping rules. The |
The Update Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET
request to the following endpoint:
curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>
When the operation state is COMPLETED
, you can verify that mTLS is enabled for the API endpoints.
Update an existing cluster to disable SASL
If you enabled mTLS and SASL on a cluster, you can disable SASL by making a PATCH /v1/clusters/{cluster.id}
request:
Show example request
CLUSTER_PATCH_BODY=`cat << EOF
{
"kafka_api": {
"sasl": {
"enabled": false
}
}
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`
Retrieve API endpoints
Retrieve the mTLS and SASL-enabled endpoints by calling the GET /v1/clusters/{id}
endpoint, passing the cluster ID as a parameter.
curl -X GET "https://api.redpanda.com/v1/clusters/<cluster-id>" \
-H "accept: application/json"\
-H "content-type: application/json" \
-H "authorization: Bearer ${AUTH_TOKEN}"
The API endpoints are returned in the response body in the following fields:
API | Field | Example |
---|---|---|
Kafka API |
|
|
HTTP Proxy |
|
|
Schema Registry |
|
|
Verify mTLS for Kafka API connections
To verify that mTLS is enabled for Kafka API, run the following rpk
command without providing a security certificate or private key:
rpk cluster info --tls-enabled
You should get the following error:
unable to request metadata: remote error: tls: certificate required
When you consume, produce to, or manage topics using rpk
, you must provide a client certificate and private key. You may use the --tls-cert
and --tls-key
options, or environment variables with rpk
.
rpk topic create test-topic --tls-enabled --tls-cert=/path/to/tls.crt --tls-key=/path/to/tls.key
Verify mTLS for HTTP Proxy and Schema Registry
To verify that mTLS is enabled for the HTTP Proxy and Schema Registry, run the following curl
commands, without providing a security certificate or key:
# Run the following to verify HTTP Proxy
curl -u $USERNAME:$PASSWORD -k -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 -d '{"records":[{"test":"hello"},{"test":"world"}]}' $HTTP_PROXY_MTLS_URL/topics/<topic-name>
# Run the following to verify Schema Registry
curl -u $USERNAME:$PASSWORD -k -H "Content-Type: application/vnd.schemaregistry.v1+json" $SCHEMA_REGISTRY_MTLS_URL/subjects/<subject-name>/versions/1
You should get an error indicating that the certificate is required.
To successfully connect to the HTTP Proxy and Schema Registry, you must provide a client certificate and private key. The following curl
commands show example requests to mTLS-enabled endpoints using test
as the username and 12345
as the password.
# HTTP Proxy
curl -u test:12345 -k --cert cert.pem --key key.pem -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 https://pandaproxy-45f811b1.cge5asc6006u7fvep0q0.fmc.dev.cloud.redpanda.com:30082/topics
# Schema Registry
curl -u test:12345 -k --cert cert.pem --key key.pem https://schema-registry-15d24f32.cge5asc6006u7fvep0q0.fmc.dev.cloud.redpanda.com:30081/subjects/Kafka-value/versions/1