# Query Iceberg Topics using AWS Glue > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Query Iceberg Topics using AWS Glue page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-redpanda-tag: v26.1.9 latest-console-tag: v3.7.3 latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: March 31, 2027 latest-connect-version: 4.93.0 docname: iceberg/iceberg-topics-aws-glue page-component-name: streaming page-version: "26.1" page-component-version: "26.1" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-topics-aws-glue.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc description: Add Redpanda topics as Iceberg tables that you can query from AWS Glue Data Catalog. # Beta release status page-beta: "true" page-git-created-date: "2025-07-30" page-git-modified-date: "2026-04-24" support-status: supported release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. This guide walks you through querying Redpanda topics as Iceberg tables stored in AWS S3, using a catalog integration with [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#data-catalog-intro). For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/). ## [](#prerequisites)Prerequisites - An AWS account with access to [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html). - Redpanda version 25.1.7 or later. - [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) installed or updated to the latest version. - [Object storage configured](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You also use the S3 bucket URI to set the base location for AWS Glue Data Catalog. - Admin permissions to create IAM policies and roles in AWS. ## [](#limitations)Limitations ### [](#lowercase-field-names-required)Lowercase field names required Use only lowercase field names. AWS Glue converts all table column names to lowercase, and Redpanda requires exact column name matches to manage schemas. Using uppercase letters prevents Redpanda from finding matching columns, which breaks schema management. ### [](#nested-partition-spec-support)Nested partition spec support AWS Glue does not support partitioning on nested fields. If Redpanda detects that the default partitioning `(hour(redpanda.timestamp))` based on the record metadata is in use, it will instead apply an empty partition spec `()`, which means the table will not be partitioned. To use partitioning, you must implement custom partitioning using your own partition columns (that is, columns that are not nested). > 📝 **NOTE** > > In Redpanda versions 25.2.1 and earlier, an empty partition spec `()` can cause a known issue that prevents certain engines like Amazon Redshift from successfully querying the table. To resolve this issue, specify custom partitioning, or upgrade Redpanda to versions 25.2.2 or later. ### [](#manual-deletion-of-iceberg-tables)Manual deletion of Iceberg tables The AWS Glue catalog integration does not support automatic deletion of Iceberg tables from Redpanda. To manually delete Iceberg tables in AWS Glue, you must either: - Set the cluster property `[iceberg_delete](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_delete)` to `false` when you configure the catalog integration. - Override the cluster property `iceberg_delete` by setting the topic property [`redpanda.iceberg.delete`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-delete) to `false` for the topic you want to delete. When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to `false`, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If you plan to re-create the topic after deleting it, you must delete the table data entirely before re-creating the topic. ## [](#authorize-access-to-aws-glue)Authorize access to AWS Glue You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog. For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources: - Root catalog (`catalog`) - All databases (`database/*`) - All tables (`table/*/*`) Your IAM policy should include a statement similar to the following: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:*" ], "Resource": [ "arn:aws:glue:::catalog", "arn:aws:glue:::database/*", "arn:aws:glue:::table/*/*" ] } ] } ``` For more information on configuring IAM permissions, see the [AWS Glue documentation](https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html). ## [](#configure-authentication-and-credentials)Configure authentication and credentials You must configure credentials for the AWS Glue Data Catalog integration in either of the following ways: - Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. This is the recommended approach. - If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: - `[iceberg_rest_catalog_credentials_source](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_credentials_source)` overrides `[cloud_storage_credentials_source](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_storage_credentials_source)` - `[iceberg_rest_catalog_aws_access_key](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_access_key)` overrides `[cloud_storage_access_key](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_storage_access_key)` - `[iceberg_rest_catalog_aws_secret_key](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_secret_key)` overrides `[cloud_storage_secret_key](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_storage_secret_key)` - `[iceberg_rest_catalog_aws_region](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_region)` overrides `[cloud_storage_region](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_storage_region)` ## [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with the AWS Glue Data Catalog: 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. Because AWS Glue provides a single catalog per account, each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. To set a unique namespace, also set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` when you set `iceberg_enabled`. This property cannot be changed after Iceberg is enabled. Run `rpk cluster config edit` to update these properties: ```bash iceberg_enabled: true # Set a custom namespace instead of the default "redpanda" iceberg_default_catalog_namespace: [""] # Glue requires Redpanda Iceberg tables to be manually deleted iceberg_delete: false iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://glue..amazonaws.com/iceberg iceberg_rest_catalog_authentication_mode: aws_sigv4 # Because Redpanda does not support the use of distinct buckets for Iceberg, # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket iceberg_rest_catalog_base_location: s3:/// # Use the iceberg_rest_catalog_aws_* properties if you want to # use separate AWS credentials for the catalog, or omit these lines to reuse S3 # (cloud_storage_*) credentials. # For access using access keys only, use iceberg_rest_catalog_aws_access_key # and iceberg_rest_catalog_aws_secret_key. For access with an IAM role, use # iceberg_rest_catalog_credentials_source only. # iceberg_rest_catalog_aws_region: # iceberg_rest_catalog_aws_access_key: # iceberg_rest_catalog_aws_secret_key: # iceberg_rest_catalog_credentials_source: ``` Use your own values for the following placeholders: - ``: A unique namespace for this cluster’s Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace `redpanda` is used. - ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your `[cloud_storage_region](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_storage_region)` or `[iceberg_rest_catalog_aws_region](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_region)` property. - `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data. `` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster. As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with your cluster’s data in object storage. ```bash Successfully updated configuration. New configuration version is 2. ``` 2. If you change the configuration for a running cluster, you must restart that cluster now. 3. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 4. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` You should see the topic as a table with data in AWS Glue Data Catalog. The data may take some time to become visible, depending on your `[iceberg_target_lag_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_target_lag_ms)` setting. 1. In AWS Glue Studio, go to Databases. 2. Select the `redpanda` database. The `redpanda` database and the table within are automatically added for you. The table name is the same as the topic name. ## [](#query-iceberg-table)Query Iceberg table You can query the Iceberg table using different engines, such as Amazon Athena, PyIceberg, or Apache Spark. To query the table or view the table data in AWS Glue, ensure that your account has the necessary permissions to access the catalog, database, and table. To query the table in Amazon Athena: 1. On the list of tables in AWS Glue Studio, click "Table data" under the **View data** column. 2. Click "Proceed" to be redirected to the Athena query editor. 3. In the query editor, select AwsDataCatalog as the data source, and select the `redpanda` database. If you set a custom namespace for your cluster, select that database instead of `redpanda`. 4. The SQL query editor should be pre-populated with a query that selects 10 rows from the Iceberg table. Run the query to see a preview of the table data. ```sql SELECT * FROM "AwsDataCatalog"."redpanda"."" limit 10; ``` Your query results should look like the following: ```sql +-----------------------------------------------------+----------------+ | redpanda | value | +-----------------------------------------------------+----------------+ | {partition=0, offset=0, timestamp=2025-07-21 | 77 6f 72 6c 64 | | 18:11:25.070000, headers=null, key=[B@1900af31} | | +-----------------------------------------------------+----------------+ ``` ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs)