# About Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: About Iceberg Topics latest-operator-version: v26.1.4 latest-console-tag: v3.7.3 latest-connect-version: 4.93.0 latest-redpanda-tag: v26.1.9 docname: iceberg/about-iceberg-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/about-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/about-iceberg-topics.adoc description: Learn how Redpanda can integrate topics with Apache Iceberg. page-git-created-date: "2025-04-04" page-git-modified-date: "2026-05-26" --- The Apache Iceberg integration for Redpanda allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems, including data warehouses like Snowflake, Databricks, ClickHouse, and Redshift, without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, such as Apache Spark and Flink, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines. Redpanda supports [version 2](https://iceberg.apache.org/spec/#format-versioning) of the Iceberg table format. ## [](#iceberg-concepts)Iceberg concepts [Apache Iceberg](https://iceberg.apache.org) is an open source format specification for defining structured tables in a data lake. The table format lets you quickly and easily manage, query, and process huge amounts of structured and unstructured data. This is similar to the way you would manage and run SQL queries against relational data in a database or data warehouse. The open format lets you use many different languages, tools, and applications to process the same data in a consistent way, so you can avoid vendor lock-in. This data management system is also known as a _data lakehouse_. In the Iceberg specification, tables consist of the following layers: - **Data layer**: Stores the data in data files. The Iceberg integration currently supports the Parquet file format. Parquet files are column-based and suitable for analytical workloads at scale. They come with compression capabilities that optimize files for object storage. - **Metadata layer**: Stores table metadata separately from data files. The metadata layer allows multiple writers to stage metadata changes and apply updates atomically. It also supports database snapshots, and time travel queries that query the database at a previous point in time. - Manifest files: Track data files and contain metadata about these files, such as record count, partition membership, and file paths. - Manifest list: Tracks all the manifest files belonging to a table, including file paths and upper and lower bounds for partition fields. - Metadata file: Stores metadata about the table, including its schema, partition information, and snapshots. Whenever a change is made to the table, a new metadata file is created and becomes the latest version of the metadata in the catalog. For Iceberg-enabled topics, the manifest files are in JSON format. - **Catalog**: Contains the current metadata pointer for the table. Clients reading and writing data to the table see the same version of the current state of the table. The Iceberg integration supports two [catalog integration](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/) types. You can configure Redpanda to catalog files stored in the same object storage bucket or container where the Iceberg data files are located, or you can configure Redpanda to use an [Iceberg REST catalog](https://iceberg.apache.org/terms/#decoupling-using-the-rest-catalog) endpoint to update an externally-managed catalog when there are changes to the Iceberg data and metadata. ![Redpanda’s Iceberg integration](https://docs.redpanda.com/cloud-data-platform/shared/_images/iceberg-integration-optimized.png) When you enable the Iceberg integration for a Redpanda topic, Redpanda brokers store streaming data in the Iceberg-compatible format in Parquet files in object storage, in addition to the log segments uploaded using Tiered Storage. Storing the streaming data in Iceberg tables in the cloud allows you to derive real-time insights through many compatible data lakehouse, data engineering, and business intelligence [tools](https://iceberg.apache.org/vendors/). ## [](#prerequisites)Prerequisites To enable Iceberg for Redpanda topics, you must have the following: - A running [BYOC](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) or BYOVPC cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC and BYOVPC, and the cluster properties to configure Iceberg are available with v25.1. - rpk: See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). - Familiarity with the Redpanda Cloud API. You must [authenticate](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication) to the Cloud API and use the Control Plane API to update your cluster configuration. ## [](#limitations)Limitations - It is not possible to append topic data to an existing Iceberg table that is not created by Redpanda. - If you enable the Iceberg integration on an existing Redpanda topic, Redpanda does not backfill the generated Iceberg table with topic data. - JSON schemas are supported starting with Redpanda version 25.2. ## [](#enable-iceberg-integration)Enable Iceberg integration To create an Iceberg table for a Redpanda topic, you must set the cluster configuration property `[iceberg_enabled](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_enabled)` to `true`, and also configure the topic property `redpanda.iceberg.mode`. You can choose to provide a schema if you need the Iceberg table to be structured with defined columns. 1. Set the `iceberg_enabled` configuration option on your cluster to `true`. When multiple clusters write to the same catalog, each cluster must use a distinct namespace to avoid table name collisions. This is especially critical for REST catalog providers that offer a single global catalog per account (such as AWS Glue), where there is no other isolation mechanism. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. To use a unique namespace for your cluster’s REST catalog integration, also set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` when you set `iceberg_enabled`. You cannot change this property after you enable Iceberg topics on the cluster. #### rpk ```bash rpk cloud login rpk profile create --from-cloud rpk cluster config set iceberg_enabled true # Optional: set a custom namespace (default is "redpanda") # rpk cluster config set iceberg_default_catalog_namespace '[""]' ``` #### Cloud API ```bash # Store your cluster ID in a variable export RP_CLUSTER_ID= # Retrieve a Redpanda Cloud access token export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "content-type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret="` # Update cluster configuration to enable Iceberg topics # Optional: to set a custom namespace (default is "redpanda"), # add "iceberg_default_catalog_namespace":[""] to custom_properties curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"cluster_configuration":{"custom_properties": {"iceberg_enabled":true}}}' ``` The [`PATCH /clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request returns the ID of a long-running operation. The operation may take up to ten minutes to complete. You can check the status of the operation by polling the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint. 2. (Optional) Create a new topic. ```bash rpk topic create ``` ```bash TOPIC STATUS OK ``` 3. Configure `redpanda.iceberg.mode` for the topic. You can choose one of the following [Iceberg modes](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/): - `key_value`: Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. - `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for this topic, with columns corresponding to each field. You must register a schema in the Schema Registry (see next step), and producers must write to the topic using the Schema Registry wire format. - `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. - `disabled` (default): Disables writing to an Iceberg table for this topic. ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` ```bash TOPIC STATUS OK ``` 4. Register a schema for the topic. This step is required for the `value_schema_id_prefix` and `value_schema_latest` modes. ```bash rpk registry schema create --schema --type ``` ```bash SUBJECT VERSION ID TYPE 1 1 PROTOBUF ``` ### [](#access-iceberg-data)Access Iceberg data To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows: | Cloud provider | Bucket or container name | Iceberg table location | | --- | --- | --- | | AWS | redpanda-cloud-storage- | redpanda-iceberg-catalog/redpanda/ | | Azure | The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. | | GCP | redpanda-cloud-storage- | For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster’s allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. For example, to add subnet IDs to the allow list through the Control Plane API [`PATCH /v1/clusters/`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) endpoint, run: ```bash curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d @- << EOF { "cloud_storage": { "azure": { "allowed_subnet_ids": [ ] } } } EOF ``` As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to [read the Iceberg topic data](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) in a data lake as you would for a relational database. See also: [Schema types translation](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#schema-types-translation). ### [](#iceberg-data-retention)Iceberg data retention Data in an Iceberg-enabled topic is consumable from Kafka based on the configured [topic retention policy](https://docs.redpanda.com/cloud-data-platform/develop/topics/create-topic/). Conversely, data written to Iceberg remains queryable as Iceberg tables indefinitely. The Iceberg table persists unless you: - Delete the Redpanda topic associated with the Iceberg table. This is the default behavior set by the `[iceberg_delete](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_delete)` cluster property and the `redpanda.iceberg.delete` topic property. If you set this property to `false`, the Iceberg table remains even after you delete the topic. - Explicitly delete data from the Iceberg table using a query engine. - Disable the Iceberg integration for the topic and delete the Parquet files in object storage. The DLQ table (`~dlq`) follows the same persistence rules as the main Iceberg table. ## [](#schema-evolution)Schema evolution Redpanda supports schema evolution in accordance with the [Iceberg specification](https://iceberg.apache.org/spec/#schema-evolution). Permitted schema evolutions include reordering fields and promoting field types. When you update the schema in Schema Registry, Redpanda automatically updates the Iceberg table schema to match the new schema. For example, if you produce records to a topic `demo-topic` with the following Avro schema: schema\_1.avsc ```avro { "type": "record", "name": "ClickEvent", "fields": [ { "name": "user_id", "type": "int" }, { "name": "event_type", "type": "string" } ] } ``` ```bash rpk registry schema create demo-topic-value --schema schema_1.avsc echo '{"user_id":23, "event_type":"BUTTON_CLICK"}' | rpk topic produce demo-topic --format='%v\n' --schema-id=topic ``` Then, you update the schema to add a new field `ts`, and produce records with the updated schema: schema\_2.avsc ```avro { "type": "record", "name": "ClickEvent", "fields": [ { "name": "user_id", "type": "int" }, { "name": "event_type", "type": "string" }, { "name": "ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-millis" } ], "default": null # Default value for the new field } ] } ``` The `ts` field can be either null or a long representing epoch milliseconds. The default value is null. ```bash rpk registry schema create demo-topic-value --schema schema_2.avsc echo '{"user_id":858, "event_type":"BUTTON_CLICK", "ts":1737998723230}' | rpk topic produce demo-topic --format='%v\n' --schema-id=topic ``` Querying the Iceberg table for `demo-topic` includes the new column `ts`: ```bash +---------+--------------+--------------------------+ | user_id | event_type | ts | +---------+--------------+--------------------------+ | 858 | BUTTON_CLICK | 2025-02-26T20:05:23.230Z | | 23 | BUTTON_CLICK | NULL | +---------+--------------+--------------------------+ ``` ## [](#next-steps)Next steps - [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/) - [Tune Performance for Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-performance-tuning/) - [Troubleshoot Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting/) ## [](#suggested-reading)Suggested reading - [Understanding Apache Kafka Schema Registry](https://www.redpanda.com/blog/schema-registry-kafka-streaming#how-does-serialization-work-with-schema-registry-in-kafka)