Streaming

Migrate Iceberg Catalogs

Use the procedures in this topic when moving from the filesystem-based object_storage catalog to a managed REST catalog, or when changing between REST catalogs. By doing so, you can switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data.

This procedure guides you through pausing Iceberg translation per topic, draining pending commits to the old catalog, applying the new catalog configuration, and restarting the cluster.

After completing these steps, you will be able to:

  • Verify that a target Iceberg catalog supports your existing schemas and partition specs

  • Pause Iceberg translation and drain pending commits without losing untranslated data

  • Apply new catalog configuration and resume translation safely

Do not change iceberg_catalog_type or any other catalog cluster property in place without following this procedure. In-flight commits and untranslated data can be lost or stuck if the catalog changes mid-translation.

Prerequisites

  • Iceberg topics enabled and running on your Redpanda cluster.

  • Network connectivity from all brokers to the new catalog endpoint.

  • Credentials configured for the new catalog (REST endpoint, authentication mode, secret or token). For configuration guidance for each catalog type, see Use Iceberg Catalogs.

  • The new catalog must support the current schema and partition spec of every Iceberg topic. See Verify catalog compatibility.

Verify catalog compatibility

Before starting the migration, verify that the new catalog can host every Iceberg topic’s table with its existing schema and partition spec. If the new catalog is incompatible with your existing schemas or partition specs, already-translated Parquet files will fail to commit, and translation will stall in a state that is difficult to recover from.

The simplest validation is to manually create a test table in the new catalog with the same schema and partition spec as one of your Iceberg topics. If the create call fails, fix the partition spec or schema before migrating. Delete the test table after validation.

AWS Glue does not support partitioning on a nested field, which is Redpanda’s default partition spec for Iceberg topics. If you migrate to AWS Glue, you must change the partition spec to a Glue-compatible form before starting the migration procedure.

Run the migration

  1. Save the current retention.ms and retention.bytes values for every Iceberg topic, then set both to -1 (infinite retention):

    rpk topic alter-config <topic-name> --set retention.ms=-1 --set retention.bytes=-1

    While Iceberg translation is paused in the next step, the topic’s retention anchor on the log is released. Without infinite retention, the cluster could delete untranslated data before the migration completes.

  2. Pause Iceberg translation on every Iceberg topic by setting redpanda.iceberg.mode to disabled. Save each topic’s previous mode value so you can restore it later.

    rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=disabled

    Setting the mode to disabled stops new translation while letting already-translated data finish committing to the old catalog. For more about Iceberg modes, see Specify Iceberg Schema.

    Do not change iceberg_enabled at the cluster level. The Iceberg integration must remain enabled at the cluster level so that pending commits can drain to the old catalog.

  3. Wait for pending commits to drain. Monitor the redpanda_iceberg_pending_commit_lag metric until it reaches 0 for every Iceberg topic-partition.

    This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. A non-zero value here is expected while translation is paused, and reflects new records the cluster has not yet translated.

    If you scrape Prometheus, the following expression returns 0 only when every Iceberg-topic partition has fully drained:

    sum(redpanda_iceberg_pending_commit_lag)

  4. Apply the new catalog configuration. For example, to switch from object_storage to a REST catalog, update the catalog cluster properties:

    rpk cluster config set iceberg_catalog_type rest
rpk cluster config set iceberg_rest_catalog_endpoint <endpoint-url>
rpk cluster config set iceberg_rest_catalog_authentication_mode oauth2
# Set additional credential properties for your chosen authentication mode.

    For full guidance on setting catalog cluster properties, see Connect to a REST catalog and the individual REST catalog integration pages.

  5. Restart Redpanda. The catalog cluster properties require a restart to take effect.

    For instructions, see Perform a Rolling Restart.

  6. After the cluster comes up, check broker logs for successful catalog requests and the absence of authentication errors to verify the new catalog connection.

  7. Resume Iceberg translation by restoring redpanda.iceberg.mode on every Iceberg topic to its previous value:

    rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=<previous-mode>

  8. Restore retention.ms and retention.bytes on every Iceberg topic to the values you saved before starting the migration.

Verify the migration

After the migration completes, confirm that new data is reaching the new catalog:

  • Query an Iceberg table in your query engine using the new catalog and confirm that row counts continue to increase as your topic produces new records.

  • Check broker logs for any commit failures referencing the new catalog. Repeated failures often indicate a schema or partition spec mismatch. See Troubleshooting for details.

Troubleshooting

  • Pending commits stuck after restart: A schema or partition spec mismatch between the original tables and the new catalog is the most common cause. See Verify catalog compatibility. If you cannot resolve the mismatch, contact Redpanda Support.

  • Authentication errors against the new REST catalog: Verify that the credential cluster properties (for example, iceberg_rest_catalog_client_id, iceberg_rest_catalog_client_secret, iceberg_rest_catalog_token) match what the new catalog expects. For OAuth, also check iceberg_rest_catalog_oauth2_server_uri.

  • Translation does not resume after restoring redpanda.iceberg.mode: Check that redpanda_iceberg_pending_translation_lag is increasing as new records are produced. If it remains 0, the cluster is not translating new records. Verify that your producer is still writing to the topic and that the topic’s mode value is one of key_value, value_schema_id_prefix, or value_schema_latest.