# Use Iceberg Topics with GCP Lakehouse > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Use Iceberg Topics with GCP Lakehouse latest-redpanda-tag: v26.1.14 latest-console-tag: v3.9.0 latest-operator-version: v26.2.1 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: March 31, 2027 latest-connect-version: 4.104.0 docname: iceberg/iceberg-topics-gcp-biglake page-component-name: streaming page-version: "26.1" page-component-version: "26.1" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-topics-gcp-biglake.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/v/26.1/modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc description: Add Redpanda topics as Iceberg tables to Google Lakehouse for Apache Iceberg that you can query from Google BigQuery. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-07-09" support-status: supported --- > ๐Ÿ“ **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/26.1/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/26.1/get-started/licensing/#self-managed) apply. > ๐Ÿ’ก **TIP** > > This guide is for integrating Iceberg topics with a managed REST catalog. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the [filesystem-based catalog](https://docs.redpanda.com/streaming/26.1/manage/iceberg/use-iceberg-catalogs/#object-storage). For an example of using the filesystem-based catalog to access Iceberg topics, see the [Getting Started with Iceberg Topics on Redpanda BYOC](https://www.redpanda.com/blog/iceberg-topics-redpanda-cloud-byoc-setup) blog post. > > The blog post uses a Redpanda Cloud cluster, but you follow the same steps for a Self-Managed cluster. This guide walks you through querying Redpanda topics as Iceberg tables stored in Google Cloud Storage, using a REST catalog integration with [Google Lakehouse for Apache Iceberg](https://docs.cloud.google.com/lakehouse/docs/introduction) (formerly BigLake). After completing this guide, you will be able to: - Create a catalog in GCP Lakehouse for Iceberg topic data. - Configure a Redpanda cluster to use GCP Lakehouse as an Iceberg REST catalog. - Query Iceberg topic data from Google BigQuery. This guide also includes optional steps to deploy a Redpanda quickstart cluster on a GCP VM instance using Docker Compose, which you can use to quickly test the Lakehouse Iceberg integration. For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/26.1/manage/iceberg/use-iceberg-catalogs/). > ๐Ÿ“ **NOTE** > > Check the [Lakehouse product page](https://docs.cloud.google.com/lakehouse/docs) for the latest status and availability of the REST Catalog API. ## [](#prerequisites)Prerequisites - A Google Cloud Platform (GCP) project. If you do not have permissions to manage GCP resources such as VMs, storage buckets, and service accounts in your project, ask your project owner to create or update them for you. - The [`gcloud` CLI](https://docs.cloud.google.com/sdk/docs/install) installed and configured for your GCP project. - [Lakehouse (BigLake) API](https://cloud.google.com/biglake/docs/enable-biglake-api) enabled for your GCP project. - Redpanda version 25.3 or later. Your Redpanda cluster must be deployed on GCP VMs. - `rpk` [installed or updated](https://docs.redpanda.com/streaming/26.1/get-started/rpk-install/) to the latest version. - [Object storage configured](https://docs.redpanda.com/streaming/26.1/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/26.1/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You also use the GCS bucket URI to set the warehouse location for the Lakehouse catalog. ## [](#limitations)Limitations ### [](#multi-region-bucket-support)Multi-region bucket support The Lakehouse runtime catalog does not support multi-region buckets. Use single-region buckets to store your Iceberg topics. ### [](#catalog-deletion)Catalog deletion Currently, it is not possible to delete non-empty Lakehouse Iceberg catalogs through the Lakehouse interface. If you need to reconfigure your setup, create a new bucket or use the REST API to remove the existing catalog. ### [](#topic-names)Topic names Lakehouse does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through Lakehouse, either: - Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`). - Ensure that the new topic names do not include dots. You must also set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). See [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) for the list of cluster properties to set when enabling the Lakehouse REST catalog integration. ## [](#set-up-google-cloud-resources)Set up Google Cloud resources ### [](#create-a-service-account-for-redpanda)Create a service account for Redpanda If you donโ€™t already have a Google Cloud service account to use, create a new service account that will be used by the VMs running Redpanda. Redpanda uses this account for writing data to Tiered Storage, Iceberg data and metadata, and for interacting with the Lakehouse catalog: ```bash gcloud iam service-accounts create --display-name "" ``` Replace the placeholder values: - ``: You can use a [name](https://docs.cloud.google.com/iam/docs/service-accounts-create) that contains lowercase alphanumeric characters and dashes. - ``: Enter a display name for the service account. ### [](#grant-required-permissions)Grant required permissions Grant the necessary permissions to your service account. To run the following commands, replace the placeholder values: - ``: The name of your service account. - ``: The name of your storage bucket. 1. Grant the service account the [Storage Object Admin role](https://docs.cloud.google.com/storage/docs/access-control/iam-roles) to access the bucket: ```bash gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" ``` 2. Grant [Service Usage Consumer](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage) and [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake#biglake.editor) roles for using the Iceberg REST catalog: ```bash gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/serviceusage.serviceUsageConsumer" gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/biglake.editor" ``` ### [](#create-a-lakehouse-catalog)Create a Lakehouse catalog Create a Lakehouse Iceberg REST catalog using the [`gcloud biglake`](https://docs.cloud.google.com/sdk/gcloud/reference/biglake/iceberg/catalogs/create) command: ```bash gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project= ``` Replace the placeholder values: - ``: Use the name of your storage bucket as the catalog ID. - ``: Your GCP project ID. ## [](#optional-deploy-redpanda-quickstart-on-gcp)Optional: Deploy Redpanda quickstart on GCP If you want to quickly test Iceberg topics in Lakehouse, you can deploy a test environment using the Redpanda Streaming quickstart. You create a new storage bucket for Tiered Storage and Iceberg data, configure a Redpanda cluster for the Lakehouse catalog integration, and deploy the cluster on a GCP Linux VM instance using Docker Compose. > ๐Ÿ“ **NOTE** > > If you already have a Redpanda cluster deployed on GCP, skip to [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg). ### [](#create-a-storage-bucket)Create a storage bucket Create a new Google Cloud Storage bucket to store Iceberg data: ```bash gcloud storage buckets create gs:// --location= ``` Replace the placeholder values: - ``: A globally unique name for your bucket. - ``: The region where you want to create the bucket, for example, `europe-west2`. > ๐Ÿ“ **NOTE** > > Ensure that the service account you created earlier has the [required permissions](#grant-required-permissions) to access this bucket. ### [](#create-vm-instances)Create VM instances Create a VM instance to run Redpanda: ```bash gcloud compute instances create \ --zone= \ --machine-type=e2-medium \ --service-account=@$(gcloud config get-value project).iam.gserviceaccount.com \ --scopes=https://www.googleapis.com/auth/cloud-platform \ --create-disk=auto-delete=yes,boot=yes,device-name=,image=projects/debian-cloud/global/images/debian-12-bookworm-v20251014,mode=rw,size=20,type=pd-standard ``` Replace the placeholder values: - ``: A name for your VM instance. - ``: The name of the service account you created earlier. - ``: The fully-qualified zone name, for example, `europe-west2-a`. ### [](#install-and-configure-redpanda)Install and configure Redpanda 1. Connect to your VM instance. It may take a few moments for the instance to be ready to accept SSH connections: ```bash gcloud compute ssh --zone ``` 2. Install Docker and Docker Compose following the [Docker installation guide](https://docs.docker.com/engine/install/debian/) for Debian. ```bash # Add Docker's official GPG key: sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update # Install Docker Engine, CLI, and Compose sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` 3. Download the Redpanda Streaming quickstart files: ```bash mkdir redpanda-quickstart && cd redpanda-quickstart && \ (1) curl -sSL https://docs.redpanda.com/26.1-redpanda-quickstart.tar.gz | tar xzf - && \ (2) cd docker-compose (3) ``` | 1 | Create and navigate to the redpanda-quickstart directory. | | --- | --- | | 2 | Download and extract the archive. | | 3 | Navigate to the Docker Compose configuration directory. | 4. Edit the `bootstrap.yaml` file to enable Tiered Storage and Iceberg features. Add or modify these sections: ```yaml # Enable Tiered Storage cloud_storage_enabled: true cloud_storage_region: n/a # GCP does not require region to be set cloud_storage_api_endpoint: storage.googleapis.com cloud_storage_api_endpoint_port: 443 cloud_storage_disable_tls: false cloud_storage_bucket: cloud_storage_credentials_source: gcp_instance_metadata # Configure Iceberg REST catalog integration with Lakehouse iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://biglake.googleapis.com/iceberg/v1/restcatalog iceberg_rest_catalog_oauth2_server_uri: https://oauth2.googleapis.com/token iceberg_rest_catalog_authentication_mode: gcp iceberg_rest_catalog_warehouse: gs:/// iceberg_rest_catalog_gcp_user_project: iceberg_dlq_table_suffix: _dlq ``` - Replace `` with your bucket name and `` with your Google Cloud project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/streaming/26.1/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). > ๐Ÿ“ **NOTE** > > If you edit `bootstrap.yml`, you can skip the cluster configuration step in [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) and proceed to the next step in that section to enable Iceberg for a topic. 5. Start Redpanda: ```bash docker compose up -d ``` 6. Install and configure `rpk`: ```bash sudo apt-get install unzip curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ rpk profile create quickstart --from-profile rpk-profile.yaml ``` ## [](#configure-redpanda-for-iceberg)Configure Redpanda for Iceberg 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. Run `rpk cluster config edit` to update these properties: ```yaml iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://biglake.googleapis.com/iceberg/v1/restcatalog iceberg_rest_catalog_oauth2_server_uri: https://oauth2.googleapis.com/token iceberg_rest_catalog_authentication_mode: gcp iceberg_rest_catalog_warehouse: gs:/// iceberg_rest_catalog_gcp_user_project: iceberg_dlq_table_suffix: _dlq ``` - Replace `` with your bucket name and `` with your Google Cloud project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/streaming/26.1/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). 2. If you change the configuration for a running cluster, you must restart that cluster now. 3. Enable the REST catalog integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/streaming/26.1/get-started/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the recordโ€™s value. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/26.1/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` > ๐Ÿ“ **NOTE** > > If youโ€™re using the Self-managed quickstart for testing, your Redpanda cluster includes a `transactions` topic with data in it, and a sample schema in the Schema Registry. To enable Iceberg for the `transactions` topic, run: > > ```bash > rpk topic alter-config transactions --set redpanda.iceberg.mode=value_schema_latest:subject=transactions > ``` Iceberg data can take a few moments to become available in Lakehouse. ## [](#query-iceberg-topics-in-bigquery)Query Iceberg topics in BigQuery 1. Navigate to the [BigQuery console](https://console.cloud.google.com/bigquery). 2. Query your Iceberg topic using SQL. For example, to query the `transactions` topic in the quickstart cluster: ```sql SELECT * FROM `>redpanda`.transactions ORDER BY redpanda.timestamp DESC LIMIT 10 ``` Replace `` with your bucket name. Your Redpanda topic is now available as Iceberg tables in Lakehouse, allowing you to run analytics queries directly on your streaming data. ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and the BigLake catalog. To grant your end users and query engines read access to the Iceberg tables in BigQuery, see [Grant permissions for BigLake tables](https://cloud.google.com/bigquery/docs/manage-open-source-metadata#grant_permissions) in the Google Cloud documentation. ## [](#optional-clean-up-resources)Optional: Clean up resources When youโ€™re finished with the quickstart example, you can clean up the resources you created: ```bash # Delete VM instances gcloud compute instances delete --zone= # Delete the storage bucket gcloud storage buckets delete gs:// # Delete the service account gcloud iam service-accounts delete @$(gcloud config get-value project).iam.gserviceaccount.com ``` > ๐Ÿ“ **NOTE** > > Manually delete the Lakehouse catalog using the [REST API](https://docs.cloud.google.com/bigquery/docs/reference/biglake/rest/v1/projects.locations.catalogs/delete). ## [](#suggested-reading)Suggested reading - [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/26.1/manage/iceberg/use-iceberg-catalogs/) - [Query Iceberg Topics](https://docs.redpanda.com/streaming/26.1/manage/iceberg/query-iceberg-topics/) - [Google Lakehouse for Apache Iceberg documentation](https://docs.cloud.google.com/lakehouse/docs/introduction) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs)