Cloud

Use Iceberg Topics with GCP Lakehouse

This guide is for integrating Iceberg topics with a managed REST catalog. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the filesystem-based catalog. For an example of using the filesystem-based catalog to access Iceberg topics, see the Getting Started with Iceberg Topics on Redpanda BYOC blog post.

This guide walks you through querying Redpanda topics as Iceberg tables stored in Google Cloud Storage, using a REST catalog integration with Google Lakehouse for Apache Iceberg (formerly BigLake).

After completing this guide, you will be able to:

  • Create a catalog in GCP Lakehouse for Iceberg topic data.

  • Configure a Redpanda cluster to use GCP Lakehouse as an Iceberg REST catalog.

  • Query Iceberg topic data from Google BigQuery.

For general information about Iceberg catalog integrations in Redpanda, see Use Iceberg Catalogs.

Check the Lakehouse product page for the latest status and availability of the REST Catalog API.

Prerequisites

  • A Google Cloud Platform (GCP) project.

    • Lakehouse must be in the same GCP project as the cluster. Cross-project Lakehouse is not supported.

      If you do not have permissions to manage GCP resources such as VMs, storage buckets, and service accounts in your project, ask your project owner to create or update them for you.

  • The gcloud CLI installed and configured for your GCP project.

  • Lakehouse (BigLake) API enabled for your GCP project.

  • Redpanda version 25.3 or later.

  • rpk installed or updated to the latest version.

For BYOC clusters created before June 9, 2026, you must re-run rpk cloud byoc gcp apply --redpanda-id=<cluster-id> --project-id=<gcp-project-id> to enable the required API services before following this guide. This is a one-time operation.

Limitations

Multi-region bucket support

The Lakehouse runtime catalog does not support multi-region buckets. Use single-region buckets to store your Iceberg topics.

Catalog deletion

Currently, it is not possible to delete non-empty Lakehouse Iceberg catalogs through the Lakehouse interface. If you need to reconfigure your setup, create a new bucket or use the REST API to remove the existing catalog.

Topic names

Lakehouse does not support Iceberg table names that contain dots (.). When creating Iceberg topics in Redpanda that you plan to access through Lakehouse, either:

  • Use the iceberg_topic_name_dot_replacement cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, current.orders and current_orders would both map to the same table name if you set the replacement to an underscore (_).

  • Ensure that the new topic names do not include dots.

You must also set the iceberg_dlq_table_suffix property to a value that does not include dots or tildes (~). See Configure Redpanda for Iceberg for the list of cluster properties to set when enabling the Lakehouse REST catalog integration.

Set up Google Cloud resources

For BYOC clusters, the required Lakehouse IAM permissions are automatically provisioned and attached to the cluster’s service account when Iceberg is enabled with a Lakehouse endpoint. You can skip to Create a Lakehouse catalog.

For BYOVPC clusters, you must grant the required permissions to your cluster’s service account and enable the biglake.googleapis.com and bigquery.googleapis.com APIs in your GCP project.

Grant required permissions

Grant the necessary permissions to your service account. To run the following commands, replace the placeholder values:

  • <service-account-name>: The name of your service account.

  • <bucket-name>: The name of your storage bucket.

    1. Grant the service account the Storage Object Admin role to access the bucket:

      gcloud storage buckets add-iam-policy-binding gs://<bucket-name> \
  --member="serviceAccount:<service-account-name>@$(gcloud config get-value project).iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

    2. Grant Service Usage Consumer and BigLake Editor roles for using the Iceberg REST catalog:

      gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
  --member="serviceAccount:<service-account-name>@$(gcloud config get-value project).iam.gserviceaccount.com" \
  --role="roles/serviceusage.serviceUsageConsumer"

gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
  --member="serviceAccount:<service-account-name>@$(gcloud config get-value project).iam.gserviceaccount.com" \
  --role="roles/biglake.editor"

Create a Lakehouse catalog

Create a Lakehouse Iceberg REST catalog using the gcloud biglake command:

gcloud biglake iceberg catalogs create <bucket-name> --catalog-type=gcs-bucket --project=<gcp-project-id>

Replace the placeholder values:

  • <bucket-name>: Use the name of your storage bucket as the catalog ID.

  • <gcp-project-id>: Your GCP project ID.

Configure Redpanda for Iceberg

  1. Edit your cluster configuration to set the iceberg_enabled property to true, and set the catalog integration properties listed in the example below.

    Use rpk as shown in the following example, or use the Cloud API to update these cluster properties. The update might take several minutes to complete.

    rpk cloud login

rpk profile create --from-cloud <cluster-id>

rpk cluster config set \
  iceberg_enabled=true \
  iceberg_catalog_type=rest \
  iceberg_rest_catalog_endpoint=https://biglake.googleapis.com/iceberg/v1/restcatalog \
  iceberg_rest_catalog_authentication_mode=gcp \
  iceberg_rest_catalog_warehouse=gs://<bucket-name>/ \
  iceberg_rest_catalog_gcp_user_project=<gcp-project-id> \
  iceberg_dlq_table_suffix=_dlq

    • <cluster-id>: Your Redpanda cluster ID.

    • <bucket-name>: For BYOC clusters, the bucket name is redpanda-cloud-storage-<cluster-id>. For BYOVPC clusters, use the name of the object storage bucket you created as a customer-managed resource.

    • <gcp-project-id>: Your GCP project ID.

    • You must set the iceberg_dlq_table_suffix property to a value that does not include dots or tildes (~). The example above uses _dlq as the suffix for the dead-letter queue (DLQ) table.

  2. Enable the REST catalog integration for a topic by configuring the topic property redpanda.iceberg.mode. The following examples show how to use rpk to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to key_value. The key_value mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See Specify Iceberg Schema for more details on Iceberg modes.

    Create a new topic and set redpanda.iceberg.mode:
    rpk topic create <topic-name> --topic-config=redpanda.iceberg.mode=key_value
    Set redpanda.iceberg.mode for an existing topic:
    rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=key_value

    Iceberg data can take a few moments to become available in Lakehouse.

Query Iceberg topics in BigQuery

  1. Navigate to the BigQuery console.

  2. Query your Iceberg topic using SQL. For example, to query the transactions topic in the quickstart cluster:

    SELECT
    *
FROM `<bucket-name>>redpanda`.transactions
ORDER BY
    redpanda.timestamp DESC
LIMIT 10

    Replace <bucket-name> with your bucket name.

Your Redpanda topic is now available as Iceberg tables in Lakehouse, allowing you to run analytics queries directly on your streaming data.

Suggested reading