IAM Roles

For Redpanda Self-Managed clusters deployed on a public cloud platform, cloud provider IAM roles (also known as managed identities) provide a safer alternative to the less secure static credential system, which is based on access keys. With static credentials, the access key and secret key are stored in plaintext in the configuration file. IAM roles are safer because they supply a role with temporary credentials that are dynamically sourced at runtime, and only last for the duration of a single session. These credentials allow you to access the data stored in an S3 bucket or Google Cloud Storage, as well as other resources.

You can use IAM roles with any Redpanda feature that makes use of cloud storage, such as Tiered Storage or Remote Read Replicas.

IAM roles and managed identities can only be configured for clusters deployed on a public cloud platform, such as Amazon Web Services (AWS), Google Cloud Platform (GCP) or Microsoft Azure. You cannot use IAM roles with on-premises clusters, even if you are using a feature that makes use of cloud storage. For on-premises clusters, you must use static access keys.

Prerequisites

Before you can configure IAM roles in Redpanda, you must create a cloud storage bucket and create an IAM policy that will be used to access that bucket. An IAM policy specifies which operations can be performed, such as writing to and reading from a cloud storage bucket, and which resources can be accessed.

AWS prerequisites

If you are using Amazon Web Services (AWS) as your cloud provider, you must satisfy the following prerequisites:

  1. Create an S3 storage bucket.

  2. Create an IAM policy.

  3. Create an IAM role and assign the policy to that role.

  4. Tiered Storage with AWS requires that the user have the following permissions to read and create objects on the bucket to be used with the cluster (or on all buckets): GetObject, DeleteObject, PutObject, PutObjectTagging, ListBucket.

  5. Bind the VM, or Pod in the case of Kubernetes, to the IAM role.

GetObject, DeleteObject, PutObject, PutObjectTagging, and ListBucket are required to fully utilize cloud storage. If the bucket is dedicated to Redpanda, we recommend allowing all (*) actions within the bucket.

Sample full access IAM policy

The following example policy grants full access to the test S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
	        "arn:aws:s3:::test",
	        "arn:aws:s3:::test/*"
	    ]
        }
    ]
}

Sample minimum read/write IAM policy

This policy represents a minimum IAM policy for the test S3 buckets required when using Tiered Storage.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::test/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::test"
        }
    ]
}

Sample read-only IAM policy

A more restrictive "read-only" IAM policy is shown below. This policy only allows a user to get and list objects in the test S3 bucket. Such a policy could be used for a read replica topic on a remote cluster that hosts read replica topics, but not Tiered Storage topics.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
	        "arn:aws:s3:::test/*"
	    ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::test"
        }
    ]
}

GCP prerequisites

If you are using Google Cloud Platform as your cloud provider, you must satisfy the following prerequisites:

  1. Create a storage bucket.

  2. Create an IAM policy that specifies the principal, the role, and the role binding. For examples, see the Google Cloud documentation.

A full access policy with all storage bucket permissions is required for Tiered Storage.

Azure prerequisites

If you are using Microsoft Azure as your cloud provider, you must satisfy the following prerequisites:

  1. Create a user-assigned managed identity^.

  2. Create an Azure storage account^.

  3. Create a container^ in the storage account.

  4. Create a custom role that only has the minimum permissions required for Tiered Storage, and assign the role to the identity. This helps prevent unauthorized actions on your data and minimize security risks. The custom role should have the following set of permissions:

    "permissions": [
    {
        "actions": [
        ],
        "notActions": [],
        "dataActions": [
            "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
            "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
            "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
            "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action",
            "Microsoft.Storage/storageAccounts/fileServices/fileShares/files/write",
            "Microsoft.Storage/storageAccounts/fileServices/writeFileBackupSemantics/action"
        ],
        "notDataActions": []
    }
]

    Assign the identity either during the creation of the storage account, or for an existing storage account. See the Azure Managed Identities documentation for more guidance.

Configuring IAM roles

After satisfying the prerequisites for your cloud platform, edit the Redpanda cluster configuration by running rpk cluster config edit. Set the cloud_storage_credentials_source property to the appropriate value for your use case. The following table shows all possible values and their descriptions.

Value Description

config_file (default)

If IAM roles are not available, specify credentials in the cluster configuration file.

aws_instance_metadata

For an AWS EC2 instance, use the instance metadata API from AWS.

sts

For AWS on Kubernetes, use the Secure Token Service (STS).

gcp_instance_metadata

For a VM running on GCP, or for Google Kubernetes Engine (GKE), use the instance metadata API from GCP.

azure_vm_instance_metadata

For a VM running on Azure, use the Instance Metadata Service (IMDS) from Azure.

azure_aks_oidc_federation

For Azure Kubernetes Service (AKS), use OIDC Issuer from Azure.

Suggested labs

Search all labs