Cloud

Query Iceberg Topics using AWS Glue

This guide walks you through querying Redpanda topics as Iceberg tables stored in AWS S3, using a catalog integration with AWS Glue. For general information about Iceberg catalog integrations in Redpanda, see Use Iceberg Catalogs.

After reading this page, you will be able to:

  • Authorize Redpanda to access AWS Glue Data Catalog.

  • Configure a Redpanda cluster to use AWS Glue as an Iceberg REST catalog.

  • Query Iceberg topic data in AWS Glue Data Catalog using Amazon Athena.

Prerequisites

  • An AWS account with access to AWS Glue Data Catalog.

    • AWS Glue Data Catalog must be in the same AWS account and region as the cluster.

  • Redpanda version 25.2 or later.

  • rpk installed or updated to the latest version.

  • Admin permissions to create IAM policies and roles in AWS.

Limitations

Lowercase field names required

Use only lowercase field names. AWS Glue converts all table column names to lowercase, and Redpanda requires exact column name matches to manage schemas. Using uppercase letters prevents Redpanda from finding matching columns, which breaks schema management.

Nested partition spec support

AWS Glue does not support partitioning on nested fields. If Redpanda detects that the default partitioning (hour(redpanda.timestamp)) based on the record metadata is in use, it will instead apply an empty partition spec (), which means the table will not be partitioned.

To use partitioning, you must implement custom partitioning using your own partition columns (that is, columns that are not nested).

In Redpanda versions 25.2.1 and earlier, an empty partition spec () can cause a known issue that prevents certain engines like Amazon Redshift from successfully querying the table. To resolve this issue, specify custom partitioning, or upgrade Redpanda to versions 25.2.2 or later.

Manual deletion of Iceberg tables

The AWS Glue catalog integration does not support automatic deletion of Iceberg tables from Redpanda. To manually delete Iceberg tables in AWS Glue, you must either:

  • Set the cluster property iceberg_delete to false when you configure the catalog integration.

  • Override the cluster property iceberg_delete by setting the topic property redpanda.iceberg.delete to false for the topic you want to delete.

When iceberg_delete or the topic override redpanda.iceberg.delete is set to false, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If you plan to re-create the topic after deleting it, you must delete the table data entirely before re-creating the topic.

Authorize access to AWS Glue

How you grant Glue access depends on your cluster type.

BYOC clusters

For BYOC clusters created in March 2026 or later, Redpanda automatically provisions the required AWS Glue IAM policy and attaches it to the cluster’s IAM role when you enable Iceberg. You don’t need to manually create IAM policies or roles for Glue access.

For BYOC clusters created before March 2026, you must re-run rpk cloud byoc aws apply --redpanda-id=<cluster-id> to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster’s IAM role with the necessary Glue permissions.

BYOVPC clusters

On BYOVPC clusters, Redpanda doesn’t create or attach the AWS Glue IAM policies, regardless of when you created the cluster. Running rpk cloud byoc aws apply doesn’t grant Glue access. Instead, the Redpanda BYOVPC Terraform module provides the policies, and you attach them to the cluster’s IAM roles.

  1. In your redpanda-data/redpanda-byovpc/aws module configuration, set enable_glue_iceberg_catalog = true. Use module version 2.1.16 or later. If you enable Redpanda SQL, use 2.1.19 or later, which adds a read-only Glue policy for the SQL engine.

    module "redpanda_byovpc" {
  source = "redpanda-data/redpanda-byovpc/aws"
  # ... existing module configuration ...
  enable_glue_iceberg_catalog = true
}

    The module adds the required Glue actions to the Redpanda agent’s permissions boundary. It also creates the following Glue IAM policies and exports their ARNs:

    • glue_iceberg_policy_arn: Read/write access for the brokers.

    • glue_iceberg_read_only_policy_arn: Read-only access for the Redpanda SQL engine. The module creates this policy only if you also set enable_redpanda_sql = true.

    If you enable Redpanda SQL and you set iceberg_default_catalog_namespace to a value other than the default, set the module’s glue_iceberg_namespaces variable to the same value. The read-only policy grants access only to the Glue databases in glue_iceberg_namespaces, which defaults to ["redpanda"].

  2. Run terraform init -upgrade, and then run terraform apply. If your configuration sets a module version, first change it to 2.1.16 or later, or to 2.1.19 or later if you enable Redpanda SQL.

  3. After Redpanda creates the cluster, attach the Glue policies to the cluster’s IAM roles. Redpanda creates these roles when it creates the cluster, so the module can’t attach the policies for you. Attach the policies from the Terraform workspace that creates the cluster:

    • glue_iceberg_policy_arn to redpanda-cloud-storage-manager-<cluster-id>: The brokers use this role to create and update Iceberg tables in AWS Glue.

    • glue_iceberg_read_only_policy_arn to redpanda-<cluster-id>-redpanda-oxla-cluster: If you enable Redpanda SQL, the SQL engine uses this role to read the Glue catalog when you query Iceberg topics. If you later disable Redpanda SQL, detach this policy first. Otherwise, disabling Redpanda SQL can fail. See Disable Redpanda SQL.

    Replace <cluster-id> with your cluster ID. In the example, replace redpanda_cluster.cluster with the name of your redpanda_cluster resource:

    resource "aws_iam_role_policy_attachment" "glue_storage_manager" {
  role       = "redpanda-cloud-storage-manager-${redpanda_cluster.cluster.id}"
  policy_arn = module.redpanda_byovpc.glue_iceberg_policy_arn
  depends_on = [redpanda_cluster.cluster]
}

# Only if Redpanda SQL is enabled
resource "aws_iam_role_policy_attachment" "glue_rpsql_engine" {
  role       = "redpanda-${redpanda_cluster.cluster.id}-redpanda-oxla-cluster"
  policy_arn = module.redpanda_byovpc.glue_iceberg_read_only_policy_arn
  depends_on = [redpanda_cluster.cluster]

  lifecycle {
    create_before_destroy = true
  }
}

    Without the broker role attachment, Iceberg translation fails on its first call to AWS Glue. Redpanda writes no data files and no dead-letter queue records. Without the SQL engine role attachment, REFRESH of the Glue-backed catalog fails with a permissions error.

    If you attached glue_iceberg_policy_arn to the SQL engine role before module version 2.1.19, the SQL engine has write access to every Glue database in the account and region. To switch the SQL engine to the read-only policy:

    1. If your configuration sets a module version, change it to 2.1.19 or later.

    2. Run terraform init -upgrade, and then run terraform apply. The upgraded module creates the read-only policy.

    3. Update the glue_rpsql_engine resource to match the example: set policy_arn to module.redpanda_byovpc.glue_iceberg_read_only_policy_arn, and add the lifecycle block. Then run terraform apply.

    With create_before_destroy, Terraform attaches the read-only policy before it detaches the old one, so queries keep working.

Before you delete a BYOVPC cluster, detach the Glue policies from both roles. If you use Terraform, run terraform destroy in the workspace that creates the cluster, which removes the attachments first.

Configure authentication and credentials

You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways:

  • Allow Redpanda to use the same object storage credential properties already configured for S3. This is the recommended approach. On BYOC clusters, the cluster’s existing AWS credentials already include the necessary Glue permissions. On BYOVPC clusters, they include these permissions after you attach the Glue policy.

    For an example cluster configuration that uses the same IAM credentials for both S3 and AWS Glue, see the Use cluster’s IAM credentials tab in the next section.

  • If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named iceberg_rest_catalog_aws_* that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication:

    For an example cluster configuration that uses separate access keys for AWS Glue, see the Use static credentials (override IAM) tab in the next section.

Update cluster configuration

To configure your Redpanda cluster to enable Iceberg on a topic and integrate with the AWS Glue Data Catalog:

  1. Edit your cluster configuration to set the iceberg_enabled property to true, and set the catalog integration properties listed in the example below.

    By default, Redpanda creates Iceberg tables in a namespace called redpanda. Because AWS Glue provides a single catalog per account, each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. To set a unique namespace, also set iceberg_default_catalog_namespace when you set iceberg_enabled. This property cannot be changed after Iceberg is enabled.

    Use rpk as shown in the following examples, or use the Cloud API to update these cluster properties. The update might take several minutes to complete.

    • Use cluster’s IAM credentials

    • Use static credentials (override IAM)

    # Glue requires Redpanda Iceberg tables to be manually deleted
# so iceberg_delete is set to false.
rpk cloud login

rpk profile create --from-cloud <cluster-id>

rpk cluster config set \
  iceberg_enabled=true \
  iceberg_delete=false \
  iceberg_default_catalog_namespace='["<custom-namespace>"]' \
  iceberg_catalog_type=rest \
  iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
  iceberg_rest_catalog_authentication_mode=aws_sigv4 \
  iceberg_rest_catalog_credentials_source=sts \
  iceberg_rest_catalog_aws_region=<glue-region> \
  iceberg_rest_catalog_base_location=s3://<cluster-storage-bucket-name>/<warehouse-path>
    # Glue requires Redpanda Iceberg tables to be manually deleted
# so iceberg_delete is set to false.
rpk cluster config set \
  iceberg_enabled=true \
  iceberg_delete=false \
  iceberg_default_catalog_namespace='["<custom-namespace>"]' \
  iceberg_catalog_type=rest \
  iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
  iceberg_rest_catalog_authentication_mode=aws_sigv4 \
  iceberg_rest_catalog_credentials_source=config_file \
  iceberg_rest_catalog_aws_region=<glue-region> \
  iceberg_rest_catalog_aws_access_key=<glue-access-key> \
  iceberg_rest_catalog_aws_secret_key='${secrets.<glue-secret-key-name>}' \
  iceberg_rest_catalog_base_location=s3://<cluster-storage-bucket-name>/<warehouse-path>

    Use your own values for the following placeholders:

    • <custom-namespace>: A unique namespace for this cluster’s Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace redpanda is used.

    • <glue-region>: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your iceberg_rest_catalog_aws_region property.

    • <cluster-storage-bucket-name> and <warehouse-path>: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, s3://<cluster-storage-bucket-name>/iceberg.

      • Bucket name: For BYOC clusters, the bucket name is redpanda-cloud-storage-<cluster-id>. For BYOVPC clusters, use the name of the object storage bucket you created as a customer-managed resource.

        This must be the same bucket used for your cluster’s object storage. You cannot specify a different bucket for Iceberg data.

      • Warehouse: This is a name you choose as the logical name (such as iceberg) for the warehouse represented by all Redpanda Iceberg topic data in the cluster.

        As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with the rest of your cluster’s data in object storage.

    • <glue-access-key> (static credentials only): The AWS access key ID for your Glue service account.

    • <glue-secret-key-name> (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example iceberg_rest_catalog_aws_secret_key, you must first store the secret value.

    Successfully updated configuration. New configuration version is 2.

  2. Enable the integration for a topic by configuring the topic property redpanda.iceberg.mode. The following examples show how to use rpk to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to key_value. The key_value mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See Specify Iceberg Schema for more details on Iceberg modes.

    Create a new topic and set redpanda.iceberg.mode:
    rpk topic create <topic-name> --topic-config=redpanda.iceberg.mode=key_value
    Set redpanda.iceberg.mode for an existing topic:
    rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=key_value

  3. Produce to the topic. For example,

    echo "hello world\nfoo bar\nbaz qux" | rpk topic produce <topic-name> --format='%k %v\n'

You should see the topic as a table with data in AWS Glue Data Catalog. The data may take some time to become visible, depending on your iceberg_target_lag_ms setting.

  1. In AWS Glue Studio, go to Databases.

  2. Select the redpanda database. The redpanda database and the table within are automatically added for you. The table name is the same as the topic name.

Query Iceberg table

You can query the Iceberg table using different engines, such as Amazon Athena, PyIceberg, or Apache Spark. To query the table or view the table data in AWS Glue, ensure that your account has the necessary permissions to access the catalog, database, and table.

To query the table in Amazon Athena:

  1. On the list of tables in AWS Glue Studio, click "Table data" under the View data column.

  2. Click "Proceed" to be redirected to the Athena query editor.

  3. In the query editor, select AwsDataCatalog as the data source, and select the redpanda database.

  4. The SQL query editor should be pre-populated with a query that selects 10 rows from the Iceberg table. Run the query to see a preview of the table data.

    SELECT * FROM "AwsDataCatalog"."redpanda"."<table-name>" limit 10;

    Your query results should look like the following:

    +-----------------------------------------------------+----------------+
| redpanda                                            | value          |
+-----------------------------------------------------+----------------+
| {partition=0, offset=0, timestamp=2025-07-21        | 77 6f 72 6c 64 |
| 18:11:25.070000, headers=null, key=[B@1900af31}     |                |
+-----------------------------------------------------+----------------+

Manage access for query engine users

Redpanda manages the permissions between Redpanda and the AWS Glue Data Catalog. To grant your end users and query engines (such as Amazon Athena or Apache Spark) read access to the Iceberg tables, use AWS Lake Formation to assign table-level and column-level permissions.

Suggested reading