# Use Iceberg Topics with GCP BigLake > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Use Iceberg Topics with GCP BigLake latest-redpanda-tag: v26.1.9 latest-console-tag: v3.7.3 latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: March 31, 2027 latest-connect-version: 4.93.0 docname: iceberg/iceberg-topics-gcp-biglake page-component-name: streaming page-version: "26.1" page-component-version: "26.1" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-topics-gcp-biglake.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc description: Add Redpanda topics as Iceberg tables to your Google BigLake data lakehouse that you can query from Google BigQuery. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-05-26" support-status: supported --- > ๐Ÿ“ **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. > ๐Ÿ’ก **TIP** > > This guide is for integrating Iceberg topics with a managed REST catalog. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the [filesystem-based catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/#object-storage). For an example of using the filesystem-based catalog to access Iceberg topics, see the [Getting Started with Iceberg Topics on Redpanda BYOC](https://www.redpanda.com/blog/iceberg-topics-redpanda-cloud-byoc-setup) blog post. > > The blog post uses a Redpanda Cloud cluster, but you follow the same steps for a Self-Managed cluster. This guide walks you through querying Redpanda topics as Iceberg tables stored in Google Cloud Storage, using a REST catalog integration with [Google BigLake](https://cloud.google.com/biglake/docs/introduction). In this guide, you do the following: - Create Google Cloud resources such as a storage bucket and service account - Grant permissions to the service account to access Iceberg data in the bucket - Create a catalog in BigLake - Configure the BigLake integration for your Redpanda cluster - Query the Iceberg data in Google BigQuery This guide also includes optional steps to deploy a Redpanda quickstart cluster on a GCP VM instance using Docker Compose, which you can use to quickly test the BigLake Iceberg integration. For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/). > ๐Ÿ“ **NOTE** > > Check the [BigLake product page](https://cloud.google.com/biglake) for the latest status and availability of the REST Catalog API. ## [](#prerequisites)Prerequisites - A Google Cloud Platform (GCP) project. If you do not have permissions to manage GCP resources such as VMs, storage buckets, and service accounts in your project, ask your project owner to create or update them for you. - The [`gcloud` CLI](https://docs.cloud.google.com/sdk/docs/install) installed and configured for your GCP project. - [BigLake API](https://cloud.google.com/biglake/docs/enable-biglake-api) enabled for your GCP project. - Redpanda v26.1.9 or later. Your Redpanda cluster must be deployed on GCP VMs. - `rpk` [installed or updated](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to the latest version. - [Object storage configured](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You also use the GCS bucket URI to set the warehouse location for the BigLake catalog. ## [](#limitations)Limitations ### [](#multi-region-bucket-support)Multi-region bucket support BigLake metastore does not support multi-region buckets. Use single-region buckets to store your Iceberg topics. ### [](#catalog-deletion)Catalog deletion Currently, it is not possible to delete non-empty BigLake Iceberg catalogs through the BigLake interface. If you need to reconfigure your setup, create a new bucket or use the REST API to remove the existing catalog. ### [](#topic-names)Topic names BigLake does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through BigLake, either: - Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`). - Ensure that the new topic names do not include dots. You must also set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). See [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) for the list of cluster properties to set when enabling the BigLake REST catalog integration. ## [](#set-up-google-cloud-resources)Set up Google Cloud resources ### [](#create-a-service-account-for-redpanda)Create a service account for Redpanda If you donโ€™t already have a Google Cloud service account to use, create a new service account that will be used by the VMs running Redpanda. Redpanda uses this account for writing data to Tiered Storage, Iceberg data and metadata, and for interacting with the BigLake catalog: ```bash gcloud iam service-accounts create --display-name "" ``` Replace the placeholder values: - ``: You can use a [name](https://docs.cloud.google.com/iam/docs/service-accounts-create) that contains lowercase alphanumeric characters and dashes. - ``: Enter a display name for the service account. ### [](#grant-required-permissions)Grant required permissions Grant the necessary permissions to your service account. To run the following commands, replace the placeholder values: - ``: The name of your service account. - ``: The name of your storage bucket. 1. Grant the service account the [Storage Object Admin role](https://docs.cloud.google.com/storage/docs/access-control/iam-roles) to access the bucket: ```bash gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" ``` 2. Grant [Service Usage Consumer](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage) and [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake#biglake.editor) roles for using the Iceberg REST catalog: ```bash gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/serviceusage.serviceUsageConsumer" gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/biglake.editor" ``` ### [](#create-a-biglake-catalog)Create a BigLake catalog Create a BigLake Iceberg REST catalog using the `gcloud` CLI: > ๐Ÿ“ **NOTE** > > This command is currently pre-GA and may change. Check the [gcloud reference](https://docs.cloud.google.com/sdk/gcloud/reference/alpha/biglake/iceberg/catalogs/create) for the latest information. ```bash gcloud alpha biglake iceberg catalogs create --catalog-type=gcs-bucket --project= ``` Replace the placeholder values: - ``: Use the name of your storage bucket as the catalog ID. - ``: Your GCP project ID. ## [](#optional-deploy-redpanda-quickstart-on-gcp)Optional: Deploy Redpanda quickstart on GCP If you want to quickly test Iceberg topics in BigLake, you can deploy a test environment using the Redpanda Streaming quickstart. In this section, you create a new storage bucket for Tiered Storage and Iceberg data. You configure a Redpanda cluster for the BigLake catalog integration and deploy the cluster on a GCP Linux VM instance using Docker Compose. > ๐Ÿ“ **NOTE** > > If you already have a Redpanda cluster deployed on GCP, skip to [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg). ### [](#create-a-storage-bucket)Create a storage bucket Create a new Google Cloud Storage bucket to store Iceberg data: ```bash gcloud storage buckets create gs:// --location= ``` Replace the placeholder values: - ``: A globally unique name for your bucket. - ``: The region where you want to create the bucket, for example, `europe-west2`. > ๐Ÿ“ **NOTE** > > Ensure that the service account you created earlier has the [required permissions](#grant-required-permissions) to access this bucket. ### [](#create-vm-instances)Create VM instances Create a VM instance to run Redpanda: ```bash gcloud compute instances create \ --zone= \ --machine-type=e2-medium \ --service-account=@$(gcloud config get-value project).iam.gserviceaccount.com \ --scopes=https://www.googleapis.com/auth/cloud-platform \ --create-disk=auto-delete=yes,boot=yes,device-name=,image=projects/debian-cloud/global/images/debian-12-bookworm-v20251014,mode=rw,size=20,type=pd-standard ``` Replace the placeholder values: - ``: A name for your VM instance. - ``: The name of the service account you created earlier. - ``: The fully-qualified zone name, for example, `europe-west2-a`. ### [](#install-and-configure-redpanda)Install and configure Redpanda 1. Connect to your VM instance. It may take a few moments for the instance to be ready to accept SSH connections: ```bash gcloud compute ssh --zone ``` 2. Install Docker and Docker Compose following the [Docker installation guide](https://docs.docker.com/engine/install/debian/) for Debian. ```bash # Add Docker's official GPG key: sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update # Install Docker Engine, CLI, and Compose sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` 3. Download the Redpanda Streaming quickstart files: ```bash mkdir redpanda-quickstart && cd redpanda-quickstart && \ (1) curl -sSL https://docs.redpanda.com/redpanda-quickstart.tar.gz | tar xzf - && \ (2) cd docker-compose (3) ``` | 1 | Create and navigate to the redpanda-quickstart directory. | | --- | --- | | 2 | Download and extract the archive. | | 3 | Navigate to the Docker Compose configuration directory. | 4. Edit the `bootstrap.yaml` file to enable Tiered Storage and Iceberg features. Add or modify these sections: ```yaml # Enable Tiered Storage cloud_storage_enabled: true cloud_storage_region: n/a # GCP does not require region to be set cloud_storage_api_endpoint: storage.googleapis.com cloud_storage_api_endpoint_port: 443 cloud_storage_disable_tls: false cloud_storage_bucket: cloud_storage_credentials_source: gcp_instance_metadata # Configure Iceberg REST catalog integration with BigLake iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://biglake.googleapis.com/iceberg/v1/restcatalog iceberg_rest_catalog_oauth2_server_uri: https://oauth2.googleapis.com/token iceberg_rest_catalog_authentication_mode: gcp iceberg_rest_catalog_warehouse: gs:/// iceberg_rest_catalog_gcp_user_project: iceberg_dlq_table_suffix: _dlq ``` - Replace `` with your bucket name and `` with your Google Cloud project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). > ๐Ÿ“ **NOTE** > > If you edit `bootstrap.yml`, you can skip the cluster configuration step in [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) and proceed to the next step in that section to enable Iceberg for a topic. 5. Start Redpanda: ```bash docker compose up -d ``` 6. Install and configure `rpk`: ```bash sudo apt-get install unzip curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ rpk profile create quickstart --from-profile rpk-profile.yaml ``` ## [](#configure-redpanda-for-iceberg)Configure Redpanda for Iceberg 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. Run `rpk cluster config edit` to update these properties: ```yaml iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://biglake.googleapis.com/iceberg/v1/restcatalog iceberg_rest_catalog_oauth2_server_uri: https://oauth2.googleapis.com/token iceberg_rest_catalog_authentication_mode: gcp iceberg_rest_catalog_warehouse: gs:/// iceberg_rest_catalog_gcp_user_project: iceberg_dlq_table_suffix: _dlq ``` - Replace `` with your bucket name and `` with your Google Cloud project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). 2. If you change the configuration for a running cluster, you must restart that cluster now. 3. Enable the REST catalog integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the recordโ€™s value. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` > ๐Ÿ“ **NOTE** > > If youโ€™re using the Self-managed quickstart for testing, your Redpanda cluster includes a `transactions` topic with data in it, and a sample schema in the Schema Registry. To enable Iceberg for the `transactions` topic, run: > > ```bash > rpk topic alter-config transactions --set redpanda.iceberg.mode=value_schema_latest:subject=transactions > ``` It may take a few moments for the Iceberg data to become available in BigLake. ## [](#query-iceberg-topics-in-bigquery)Query Iceberg topics in BigQuery 1. Navigate to the [BigQuery console](https://console.cloud.google.com/bigquery). 2. Query your Iceberg topic using SQL. For example, to query the `transactions` topic in the quickstart cluster: ```sql SELECT * FROM `>redpanda`.transactions ORDER BY redpanda.timestamp DESC LIMIT 10 ``` Replace `` with your bucket name. Your Redpanda topic is now available as Iceberg tables in BigLake, allowing you to run analytics queries directly on your streaming data. ## [](#optional-clean-up-resources)Optional: Clean up resources When youโ€™re finished with the quickstart example, you can clean up the resources you created: ```bash # Delete VM instances gcloud compute instances delete --zone= # Delete the storage bucket gcloud storage buckets delete gs:// # Delete the service account gcloud iam service-accounts delete @$(gcloud config get-value project).iam.gserviceaccount.com ``` > ๐Ÿ“ **NOTE** > > Manually delete the BigLake catalog using the [REST API](https://docs.cloud.google.com/bigquery/docs/reference/biglake/rest/v1/projects.locations.catalogs/delete). ## [](#suggested-reading)Suggested reading - [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/) - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) - [Google BigLake documentation](https://cloud.google.com/biglake/docs/introduction) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs)