IAM Roles

For self-hosted clusters deployed on a public cloud platform, cloud provider IAM roles provide a safer alternative to the less secure static credential system, which is based on access keys. With static credentials, the access key and secret key are stored in plaintext in the configuration file. IAM roles are safer because they supply a role with temporary credentials that are dynamically sourced at runtime, and only last for the duration of a single session. These credentials allow you to access the data stored in an S3 bucket or Google Cloud Storage, as well as other resources.

You can use IAM roles with any Redpanda feature that makes use of cloud storage, such as Tiered Storage or Remote Read Replicas.

IAM roles can only be configured for clusters deployed on a public cloud platform, such as Amazon Web Services (AWS) or Google Cloud Platform (GCP). You cannot use IAM roles with on-premises clusters, even if you are using a feature that makes use of cloud storage. For on-premises clusters, you must use static access keys.

Prerequisites

Before you can configure IAM roles in Redpanda, you must create a cloud storage bucket and create an IAM policy that will be used to access that bucket. An IAM policy specifies which operations can be performed, such as writing to and reading from a cloud storage bucket, and which resources can be accessed.

AWS prerequisites

If you are using Amazon Web Services (AWS) as your cloud provider, you must satisfy the following prerequisites:

  1. Create an S3 storage bucket.

  2. Create an IAM policy.

  3. Create an IAM role and assign the policy to that role.

  4. Tiered Storage with AWS requires that the user have the following permissions to read and create objects on the bucket to be used with the cluster (or on all buckets): GetObject, DeleteObject, PutObject, PutObjectTagging, ListBucket.

  5. Bind the VM, or Pod in the case of Kubernetes, to the IAM role.

GetObject, DeleteObject, PutObject, PutObjectTagging, and ListBucket are required to fully utilize cloud storage. If the bucket is dedicated to Redpanda, we recommend allowing all (*) actions within the bucket.

Sample full access IAM policy

The following example policy grants full access to the test S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
	        "arn:aws:s3:::test",
	        "arn:aws:s3:::test/*"
	    ]
        }
    ]
}

Sample minimum read/write IAM policy

This policy represents a minimum IAM policy for the test S3 buckets required when using Tiered Storage.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::test/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::test"
        }
    ]
}

Sample read-only IAM policy

A more restrictive "read-only" IAM policy is shown below. This policy only allows a user to get and list objects in the test S3 bucket. Such a policy could be used for a read replica topic on a remote cluster that hosts read replica topics, but not Tiered Storage topics.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
	        "arn:aws:s3:::test/*"
	    ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::test"
        }
    ]
}

GCP prerequisites

If you are using Google Cloud Platform as your cloud provider, you must satisfy the following prerequisites:

  1. Create a storage bucket.

  2. Create an IAM policy that specifies the principal, the role, and the role binding. For examples, see the Google Cloud documentation.

A full access policy with all storage bucket permissions is required for Tiered Storage.

Configuring IAM roles

After satisfying the prerequisites for your cloud platform, edit the Redpanda cluster configuration by running rpk cluster config edit. Set the cloud_storage_credentials_source property to the appropriate value for your use case. The following table shows all possible values and their descriptions.

Value Description

config_file (default)

If IAM roles are not available, specify credentials in the cluster configuration file.

aws_instance_metadata

For an AWS EC2 instance, use the instance metadata API from AWS.

sts

For AWS on Kubernetes, use the Secure Token Service (STS).

gcp_instance_metadata

For a VM running on GCP, or for Google Kubernetes Engine (GKE), use the instance metadata API from GCP.

Suggested labs

Search all labs