For self-hosted clusters deployed on a public cloud platform, cloud provider IAM roles provide a safer alternative to the less secure static credential system, which is based on access keys. With static credentials, the access key and secret key are stored in plaintext in the configuration file. IAM roles are safer because they supply a role with temporary credentials that are dynamically sourced at runtime, and only last for the duration of a single session. These credentials allow you to access the data stored in an S3 bucket or Google Cloud Storage, as well as other resources.
IAM roles can only be configured for clusters deployed on a public cloud platform, such as Amazon Web Services (AWS) or Google Cloud Platform (GCP). You cannot use IAM roles with on-premises clusters, even if you are using a feature that makes use of cloud storage. For on-premises clusters, you must use static access keys.
Before you can configure IAM roles in Redpanda, you must create a cloud storage bucket and create an IAM policy that will be used to access that bucket. An IAM policy specifies which operations can be performed, such as writing to and reading from a cloud storage bucket, and which resources can be accessed.
If you are using Amazon Web Services (AWS) as your cloud provider, you must satisfy the following prerequisites:
- Create an S3 storage bucket.
- Create an IAM policy.
- Create an IAM role and assign the policy to that role.
- Tiered Storage with AWS requires that the user have the following permissions to read and create objects on the bucket to be used with the cluster (or on all buckets):
- Bind the VM, or Pod in the case of Kubernetes, to the IAM role.
Sample full access IAM policy
The following example policy grants all permissions associated with an S3 bucket, and full access to all resources. This particular example also includes all permissions associated with
s3-object-lambda, which you can use to perform operations on the data in the S3 bucket.
Sample read-only IAM policy
A more restrictive "read-only" IAM policy is shown below. This policy only allows a user to get and list objects in the
test S3 bucket. Such a policy could be used for a read replica topic on a remote cluster that hosts read replica topics, but not Tiered Storage topics.
If you are using Google Cloud Platform as your cloud provider, you must satisfy the following prerequisites:
- Create a storage bucket.
- Create an allow policy, also called an IAM policy, that specifies the principal, the role, and the role binding.
A full access policy with all storage bucket permissions is required for Tiered Storage.
Configuring IAM roles
After satisfying the prerequisites for your cloud platform, edit the Redpanda cluster configuration by running
rpk cluster config edit. Set the
cloud_storage_credentials_source property to the appropriate value for your use case. The following table shows all possible values and their descriptions.
|If IAM roles are not available, specify credentials in the cluster configuration file.|
|For an AWS EC2 instance, use the instance metadata API from AWS.|
|For AWS on Kubernetes, use the Secure Token Service (STS).|
|For a VM running on GCP, or for Google Kubernetes Engine (GKE), use the instance metadata API from GCP.|