Connect

Tune the Iceberg output for throughput

Tune the iceberg output for high write throughput to Apache Iceberg tables. Use these two tested recipes to choose the right trade-off between preserving message ordering and maximizing throughput.

After reading this page, you will be able to:

  • Configure an order-preserving Iceberg sink that sustains high throughput

  • Configure an unordered Iceberg sink for maximum throughput

  • Choose the recipe that matches your ordering and throughput requirements

Both recipes target roughly 10,000 rows or 10 seconds per commit, whichever comes first. This keeps commit sizes large enough to amortize the cost of each Iceberg snapshot while bounding commit latency.

Prerequisites

These recipes tune append-only sinks, where every message becomes a new row. They use a high max_in_flight to commit batches concurrently. Do not apply them to keyed change-data-capture workloads that use row_operation with upsert or delete. Keyed workloads require max_in_flight: 1 to preserve per-key order. For more detail, see Row-level operations.

Recipe A: preserve message ordering

Use this recipe when you want to process records as a single ordered stream, rather than in parallel per partition.

A single merged input stream preserves the processing order of records, because Redpanda Connect consumes the stream in order instead of parallelizing consumption per partition. To keep that ordered stream fast, decouple the input from the commit-bound output and batch at the output level:

  • A memory buffer absorbs short input bursts and commit stalls, decoupling the input from commit latency. The buffer is bounded (500 MiB in this recipe): if the input sustains a higher rate than the sink can commit, the buffer fills and applies backpressure to the input. Size the buffer to your throughput multiplied by the commit interval.

  • Output-level batching feeds the Iceberg commit coalescer, so max_in_flight can dispatch concurrent writers. Without output batching, a single buffered stream starves the committer. In smoke tests at 1 vCPU, adding output batching raised throughput from 0.8 to 12.7 MB/s.

# Configure catalog, storage, namespace, and table as usual.
buffer:
  memory:
    limit: 524288000        # 500 MiB: size to throughput x commit interval
    batch_policy:
      count: 10000
      period: 10s
output:
  iceberg:
    max_in_flight: 16
    batching:
      count: 10000
      period: 10s
    commit:
      max_snapshot_age: 24h

max_in_flight: 16 lets the output commit batches concurrently for throughput. Because concurrent batches can commit in any order, the order in which rows become visible in the Iceberg table is not guaranteed to match the input order, even though the input stream itself is processed in order. If you need commits applied in strict order, set max_in_flight: 1, at the cost of throughput.

Recipe B: maximize throughput

Use this recipe when preserving processing order is not required, which is typical for append-only Iceberg sinks. This recipe reaches the highest throughput ceiling.

Recipe B removes the buffer and parallelizes at the input instead. The redpanda input's unordered_processing processes each partition’s records in parallel and batches them in the read-ahead fetch buffer, feeding many concurrent commits to the coalescer. A higher max_in_flight of 32 lets those commits overlap. This configuration trades ordered processing for higher throughput.

input:
  redpanda:
    # Configure seed_brokers, topics, and consumer_group as usual.
    unordered_processing:
      enabled: true
      checkpoint_limit: 1024
      batching:
        count: 10000
        period: 10s
output:
  iceberg:
    max_in_flight: 32

Compare recipe throughput

The following table shows mean write throughput in MB/s for each recipe as vCPUs scale. Results were measured on an Amazon Web Services (AWS) c8g.4xlarge instance against a roughly 178 GB dataset, using an AWS Glue REST catalog with Amazon S3 storage.

vCPU Recipe A (ordered) Recipe B (unordered)

1

15.9

38.3

2

69.1

64.1

4

114.2

98.6

8

109.2

122.4

When CPU is constrained, Recipe B’s per-partition parallelism gives a clear advantage. Recipe A scales strongly up to 4 vCPUs and then plateaus, while Recipe B continues to climb as more vCPUs become available.

Your results depend on your catalog, storage backend, message size, and partition count. Use these numbers as a starting point and benchmark against your own workload.

Choose a recipe

Requirement Recipe

Ordered processing is not required (typical for append-only Iceberg sinks)

Recipe B: maximize throughput

You want the stream processed in order

Recipe A: preserve message ordering

Next steps