Query Iceberg Topics using Snowflake and Open Catalog

In Open Catalog, select Connections, then + Connection.

In Configure Service Connection, provide a name. Open Catalog creates a new principal with this name.

Make sure Create new principal role is toggled on.

Enter a name for the principal role. Then, click Create.

In Open Catalog, select Catalogs, and select your catalog.

On the Roles tab of your catalog, click + Catalog Role.

Give the catalog role a name.

Under Privileges, select CATALOG_MANAGE_CONTENT. This provides full management privileges for the catalog. Then, click Create.

On the Roles tab of the catalog, click Grant to Principal Role.

Select the catalog role you just created.

Select the principal role you created earlier. Click Grant.

Edit your cluster configuration to set the iceberg_enabled property to true, and set the catalog integration properties listed in the example below. You must restart your cluster if you change this configuration for a running cluster. You can run rpk cluster config edit to update these properties:

iceberg_enabled: true
iceberg_catalog_type: rest
iceberg_rest_catalog_endpoint: https://<snowflake-orgname>-<open-catalog-account-name>.snowflakecomputing.com/polaris/api/catalog
iceberg_rest_catalog_client_id: <open-catalog-connection-client-id>
iceberg_rest_catalog_client_secret: <open-catalog-connection-client-secret>
iceberg_rest_catalog_prefix: <open-catalog-name>

# Optional
iceberg_translation_interval_ms_default: 1000
iceberg_catalog_commit_interval_ms: 1000

Successfully updated configuration. New configuration version is 2.

You must restart your cluster so that the configuration changes take effect.

Enable the integration for a topic by configuring the topic property redpanda.iceberg.mode. This mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. See Enable Iceberg integration for more details on Iceberg modes. The following examples show how to use rpk to either create a new topic, or alter the configuration for an existing topic, to set the Iceberg mode to key_value.

rpk topic create <topic-name> --topic-config=redpanda.iceberg.mode=key_value

rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=key_value

Produce to the topic. For example,

echo "hello world\nfoo bar\nbaz qux" | rpk topic produce <topic-name> --format='%k %v\n'

In Open Catalog, select Catalogs, then open your catalog.

Under your catalog, you should see the redpanda namespace, and a table with the name of your topic. The redpanda namespace and the table are automatically added for you.

Run the CREATE CATALOG INTEGRATION command in Snowflake:

CREATE CATALOG INTEGRATION <catalog-integration-name>
  CATALOG_SOURCE = POLARIS
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'redpanda'
  REST_CONFIG = (
    CATALOG_URI = '<open-catalog-uri>'
    WAREHOUSE = '<open-catalog-name>'
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_CLIENT_ID = '<open-catalog-connection-client-id>'
    OAUTH_CLIENT_SECRET = '<open-catalog-connection-client-secret>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  REFRESH_INTERVAL_SECONDS = 30
  ENABLED = TRUE;

Run the following command to verify that the catalog is integrated correctly:

SELECT SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('<catalog-integration-name>');

# Example result for redpanda.iceberg.mode=key_value
+-----------------------------------------------------------------------+
| SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('<catalog_integration_name>') |
+-----------------------------------------------------------------------+
| [{"namespace":"redpanda","name":"<table_name>"}]                      |
+-----------------------------------------------------------------------+