---
title: gcp_bigquery_write_api
latest-connect-version: 4.118.0
latest-operator-version: v26.2.6
latest-console-tag: v3.14.0
latest-redpanda-tag: v26.2.7
docname: outputs/gcp_bigquery_write_api
page-component-name: connect
page-version: master
page-component-version: master
page-component-title: Connect
page-relative-src-path: outputs/gcp_bigquery_write_api.adoc
page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_bigquery_write_api.adoc
description: Streams data into BigQuery using the Storage Write API.
page-cloud-available: "true"
page-cloud-available-url: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_bigquery_write_api/
page-git-created-date: "2026-05-11"
page-git-modified-date: "2026-10-29"
page-generated-sources:
  - include: connect:components:partial$descriptions/outputs/gcp_bigquery_write_api.adoc
    generator: null
    version: null
    source: the component specs in https://github.com/redpanda-data/connect
  - include: connect:components:partial$availability/outputs/gcp_bigquery_write_api.adoc
    generator: null
    version: null
    source: the component specs in https://github.com/redpanda-data/connect
  - include: connect:components:partial$fields/outputs/gcp_bigquery_write_api.adoc
    generator: null
    version: null
    source: the component specs in https://github.com/redpanda-data/connect
page-generated-content: partial
---

# gcp_bigquery_write_api

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [connect-full.txt](https://docs.redpanda.com/connect-full.txt)

<!-- Source: https://docs.redpanda.com/connect/components/outputs/gcp_bigquery_write_api.md -->

**License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days.

Streams data into BigQuery using the Storage Write API.

Introduced in version 4.90.0.

Writes messages to a BigQuery table using the Storage Write API. This provides higher throughput and lower latency than the legacy streaming API or load jobs.

Messages can be formatted as JSON (default) or raw protobuf bytes. When using JSON format the component automatically fetches the table schema and converts each message to the corresponding proto representation.

> ⚠️ **WARNING**
>
> protojson encodes int64 and uint64 values as strings, bytes as base64-encoded strings, and timestamps as RFC 3339 strings. JSON messages must follow these conventions (for example, `"age": "30"`, `"data": "aGVsbG8="`, `"created_at": "2026-01-02T15:04:05Z"`); otherwise the write will fail with an unmarshalling error.

When batching is enabled the table name is resolved from the first message in each batch. All messages in the same batch are written to that table.

The interpolated table name is sanitized for BigQuery: dots, hyphens, slashes and whitespace are replaced with underscores, non-ASCII-alphanumeric characters are stripped, leading digits are prefixed with `_`, and the result is truncated to 1024 characters. A name that sanitizes to the empty string is rejected as a permanent error.

## Write modes

The `write_mode` field selects between two write paths:

-   `default_stream` (default): the multiplexed default stream. Lowest latency, at-least-once semantics.

-   `pending_stream`: a fresh pending stream is allocated per batch; rows are written with sequential offsets, the stream is finalized, then atomically committed. Provides exactly-once semantics within a single committed batch.


## Auto-create

When `auto_create_table` is true, the output creates missing tables on the fly using the configured `schema`, `time_partitioning`, and `clustering`. `AlreadyExists` errors from concurrent creators are treated as success. When the table name is interpolated, every auto-created table receives the same configuration.

## Exactly-once caveat

The exactly-once guarantee of `pending_stream` is "exactly-once within a stream". If a BatchCommitWriteStreams RPC succeeds but its response is lost to a network failure, benthos retries the batch through a new pending stream and the data lands twice. This is a fundamental limitation of the BigQuery Storage Write API exactly-once contract and applies to every implementation.

## CDC migration

When migrating from the load-jobs based `gcp_bigquery` output to CDC mode, see the [CDC migration guide](https://docs.redpanda.com/connect/components/outputs/bigquery_cdc_migration/).

### Common

```yml
output:
  label: ""
  gcp_bigquery_write_api:
    project: ""
    dataset: "" # No default (required)
    table: "" # No default (required)
    message_format: json
    change_type: "" # No default (optional)
    change_sequence_number: "" # No default (optional)
    primary_keys: [] # No default (optional)
    max_in_flight: 4
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
    credentials_json: ""
```

### Advanced

```yml
output:
  label: ""
  gcp_bigquery_write_api:
    project: ""
    dataset: "" # No default (required)
    table: "" # No default (required)
    message_format: json
    write_mode: default_stream
    change_type: "" # No default (optional)
    change_sequence_number: "" # No default (optional)
    primary_keys: [] # No default (optional)
    auto_create_table: false
    schema: []
    time_partitioning:
      type: "" # No default (optional)
      field: ""
      expiration: 0s
      require_filter: false
    clustering: []
    max_in_flight: 4
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""
      processors: [] # No default (optional)
    credentials_json: ""
    target_principal: ""
    delegates: []
    stream_idle_timeout: 5m
    stream_sweep_interval: 1m
    max_cached_streams: 1024
    schema_resolve_timeout: 15s
    schema_evolution_timeout: 30s
    endpoint:
      http: ""
      grpc: ""
```

## Fields

### `auto_create_table`

If true and the target table does not exist, the output creates it using the configured `schema`, `time_partitioning`, and `clustering`. AlreadyExists errors from concurrent creators are treated as success. When the table name is interpolated, every auto-created table receives the same schema and partition/clustering settings.

Requires version 4.94.0 or later.

**Type**: `bool`

**Default**: `false`

### `batching`

Configure a [batching policy](https://docs.redpanda.com/connect/configuration/batching/).

**Type**: `object`

```yaml
# Examples:
batching:
  byte_size: 5000
  count: 0
  period: 1s

# ---

batching:
  count: 10
  period: 1s

# ---

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m
```

### `batching.byte_size`

The maximum total size (in bytes) that a batch can reach before it is flushed. When the combined size of all messages in the batch reaches or exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output).

Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `count` or `period`).

**Type**: `int`

**Default**: `0`

### `batching.check`

A [Bloblang query](https://docs.redpanda.com/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch.

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
check: this.type == "end_of_transaction"
```

### `batching.count`

The number of messages at which the batch is flushed. Set to `0` to disable count-based batching.

**Type**: `int`

**Default**: `0`

### `batching.period`

The length of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`.

**Type**: `string`

**Default**: `""`

```yaml
# Examples:
period: 1s

# ---

period: 1m

# ---

period: 500ms
```

### `batching.processors[]`

A list of [processors](https://docs.redpanda.com/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so splitting the batch into smaller batches with these processors has no effect.

**Type**: `array<processor>`

```yaml
# Examples:
processors:
  - archive:
      format: concatenate


# ---

processors:
  - archive:
      format: lines


# ---

processors:
  - archive:
      format: json_array
```

### `change_sequence_number`

Optional Bloblang expression resolving to the `_CHANGE_SEQUENCE_NUMBER` pseudo-column value. Format: 1 to 4 sections of 1 to 16 hexadecimal characters each, separated by `/`. Example: `${! metadata("scn") }` or `${! "0/0/0/0" }`. When unset, BigQuery resolves ordering by arrival time.

This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries).

Requires version 4.96.0 or later.

**Type**: `string`

### `change_type`

Bloblang expression resolving to the `_CHANGE_TYPE` pseudo-column value for each row. Must resolve to `UPSERT` or `DELETE` (case-insensitive). Required when `write_mode` is `upsert` or `upsert_delete`. Example: `${! metadata("operation") }`.

This field supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries).

Requires version 4.96.0 or later.

**Type**: `string`

### `clustering[]`

Optional clustering columns (up to 4) applied during `auto_create_table`. All names must appear in `schema`.

Requires version 4.94.0 or later.

**Type**: `array<string>`

**Default**: `[]`

### `credentials_json`

An optional JSON string containing GCP credentials. If empty, credentials are loaded from the environment.

> ⚠️ **CAUTION**
>
> This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](https://docs.redpanda.com/connect/configuration/secrets/).

**Type**: `string`

**Default**: `""`

### `dataset`

The BigQuery dataset ID.

**Type**: `string`

### `delegates[]`

Optional delegation chain for chained service account impersonation. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain.

**Type**: `array<string>`

**Default**: `[]`

### `endpoint`

Optional endpoint overrides for the BigQuery and Storage Write API clients.

**Type**: `object`

### `endpoint.grpc`

Override the BigQuery Storage gRPC endpoint. Useful for local emulators.

**Type**: `string`

**Default**: `""`

### `endpoint.http`

Override the BigQuery HTTP endpoint. Useful for local emulators.

**Type**: `string`

**Default**: `""`

### `max_cached_streams`

Soft cap on the number of cached streams. When the cache exceeds this size, the least-recently-used stream is evicted. Set to 0 for unlimited (rely on idle-timeout sweeping only). Relevant when the table field uses interpolation to route to many tables.

Requires version 4.93.0 or later.

**Type**: `int`

**Default**: `1024`

### `max_in_flight`

The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput.

**Type**: `int`

**Default**: `4`

### `message_format`

The format of input messages. Use 'json' to have the component convert JSON to proto automatically. Use 'protobuf' to supply raw proto-encoded bytes.

**Type**: `string`

**Default**: `json`

**Options**: `json`, `protobuf`

### `primary_keys[]`

Optional list of primary-key column names. Required when `auto_create_table` is true and `write_mode` is `upsert` or `upsert_delete`. A pre-existing table must already declare its PRIMARY KEY: this field cannot add one; when both are set they must match exactly (same columns, same order). Up to 16 columns; composite keys are supported in the same order they are listed.

Requires version 4.96.0 or later.

**Type**: `array<string>`

### `project`

The GCP project ID. If empty, the project is auto-detected from the environment.

**Type**: `string`

**Default**: `""`

### `schema[]`

Column definitions used by `auto_create_table`. Required when `auto_create_table` is true.

Requires version 4.94.0 or later.

**Type**: `array<object>`

**Default**: `[]`

### `schema[].fields[]`

For RECORD columns, the list of nested fields. Same shape as the top-level schema list.

**Type**: `array`

### `schema[].mode`

Column mode: NULLABLE (default), REQUIRED, or REPEATED.

**Type**: `string`

**Default**: `NULLABLE`

### `schema[].name`

Column name.

**Type**: `string`

### `schema[].type`

BigQuery column type (STRING, BYTES, INTEGER/INT64, FLOAT/FLOAT64, NUMERIC, BIGNUMERIC, BOOLEAN/BOOL, TIMESTAMP, DATE, TIME, DATETIME, GEOGRAPHY, JSON, RECORD).

**Type**: `string`

### `schema_evolution_timeout`

Total time budget for a single schema evolution attempt (Metadata + Update across all CAS retries on HTTP 412). Bounds how long the WriteBatch retry loop can be starved by a wedged backend.

Requires version 4.93.0 or later.

**Type**: `string`

**Default**: `30s`

### `schema_resolve_timeout`

How long a single BigQuery table-metadata fetch can run before being aborted. Coalesced concurrent resolves share one fetch, so this bounds the time a wedged backend can stall every batch routing to the same table. On the auto_create_table path the budget covers Metadata→Create→Metadata, so it needs to absorb transient backend slowness on top of the metadata fetch itself.

Requires version 4.93.0 or later.

**Type**: `string`

**Default**: `15s`

### `stream_idle_timeout`

How long a cached stream can remain unused before being closed. Relevant when the table field uses interpolation to route to many tables.

**Type**: `string`

**Default**: `5m`

### `stream_sweep_interval`

How often to check for idle streams to close.

**Type**: `string`

**Default**: `1m`

### `table`

The BigQuery table ID. Supports [interpolation functions](https://docs.redpanda.com/connect/configuration/interpolation/#bloblang-queries). When batching, resolved from the first message in each batch.

**Type**: `string`

### `target_principal`

Service account email to impersonate. When set, the output obtains tokens acting as this service account. Requires the caller to have roles/iam.serviceAccountTokenCreator on the target.

**Type**: `string`

**Default**: `""`

### `time_partitioning`

Optional time-partitioning settings applied during `auto_create_table`. Setting `type` is the trigger: when omitted, the block is treated as absent.

Requires version 4.94.0 or later.

**Type**: `object`

### `time_partitioning.expiration`

Optional partition expiration. Zero means no expiration.

**Type**: `string`

**Default**: `0s`

### `time_partitioning.field`

Column to partition on. Must be of type DATE, TIMESTAMP, or DATETIME. If empty, the table uses ingestion-time partitioning (`_PARTITIONTIME`).

**Type**: `string`

**Default**: `""`

### `time_partitioning.require_filter`

If true, queries against the table must filter on the partition column.

**Type**: `bool`

**Default**: `false`

### `time_partitioning.type`

Partitioning granularity.

**Type**: `string`

**Options**: `DAY`, `HOUR`, `MONTH`, `YEAR`

### `write_mode`

How the output writes to BigQuery. `default_stream` uses the multiplexed default stream (at-least-once, lowest latency). `pending_stream` allocates a per-batch pending stream that commits atomically, providing exactly-once semantics within a single committed batch. `upsert` writes UPSERT-only rows to a BigQuery CDC-enabled table; the target table must have a PRIMARY KEY. `upsert_delete` allows both UPSERT and DELETE rows. Both CDC modes use the default stream as required by BigQuery.

Requires version 4.94.0 or later.

**Type**: `string`

**Default**: `default_stream`

**Options**: `default_stream`, `pending_stream`, `upsert`, `upsert_delete`