Cloud

Enable Redpanda SQL on a BYOVPC Cluster on GCP

BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or Redpanda Sales.

Enable Redpanda SQL on a Bring Your Own Virtual Private Cloud (BYOVPC) cluster on GCP so you can query streaming data in Redpanda topics using standard PostgreSQL syntax. You can enable SQL when creating a new BYOVPC cluster or on an existing one.

Unlike standard BYOC clusters, Redpanda does not create resources in your GCP project for BYOVPC clusters. You must provision the SQL-specific GCP resources yourself and supply them to the cluster as customer-managed resources (CMRs) before enabling the SQL engine. You can enable SQL on an existing cluster from the Cloud Console, the Terraform provider, or the Cloud API. To enable SQL when you create a new cluster, use the Terraform provider or the Cloud API.

After reading this page, you will be able to:

  • Provision SQL-specific GCP resources using the Redpanda BYOVPC Terraform module

  • Enable Redpanda SQL on a new or existing BYOVPC cluster by supplying customer-managed resources

  • Verify that the SQL engine is running and ready to accept connections

Prerequisites

  • For a new cluster: a BYOVPC network on GCP. For an existing cluster: a deployed BYOVPC cluster on GCP.

  • The Redpanda BYOVPC Terraform module and Terraform version 1.8.5 or later, to provision the SQL resources.

  • If you enable SQL with Terraform: the Redpanda Terraform provider, configured with valid credentials. Enabling SQL on a new cluster requires version 2.1.1 or later; enabling on an existing cluster requires version 2.2.0 or later, because earlier versions replace the cluster when you add the SQL customer-managed resources.

  • Admin permissions in your Redpanda Cloud organization on usage-based billing. If using the Cloud API, a valid bearer token with permission to update clusters. See Authenticate to the Cloud API.

  • To verify the engine after enabling it: psql version 16 or later on a host inside the cluster’s VPC. A BYOVPC cluster is private, so the SQL endpoint is not reachable from outside the VPC.

Provision SQL resources with Terraform

In your redpanda-data/redpanda-byovpc/gcp module configuration, set enable_redpanda_sql = true:

module "redpanda_gcp" {
  source = "redpanda-data/redpanda-byovpc/gcp"
  # ... existing configuration ...
  enable_redpanda_sql = true
}

Apply the updated configuration:

terraform apply

This creates the SQL-specific GCP resources that the base BYOVPC provisioning does not, and grants them the IAM roles and Workload Identity bindings that the SQL engine needs:

  • A GCP service account for the SQL compute nodes, which reads and writes the SQL data-storage bucket.

  • A GCP service account for the SQL API, which accesses Secret Manager secrets under the SQL prefix.

  • A GCS bucket for SQL data storage.

  • A Secret Manager prefix: a string namespace (not a resource) that reserves where Redpanda SQL Iceberg catalog credentials are stored.

After applying, Terraform outputs the following values. Supply them to the cluster when you enable SQL.

Module output Description

redpanda_sql_service_account_email

Email of the SQL compute-node service account

redpanda_sql_api_service_account_email

Email of the SQL API service account

redpanda_sql_storage_bucket_name

Name of the GCS bucket for SQL data storage

redpanda_sql_secret_manager_prefix

Secret Manager prefix for SQL Iceberg catalog credentials

Enable Redpanda SQL

Use the Cloud Console (existing clusters only), the Terraform provider, or the Cloud API to enable the SQL engine. The steps differ depending on whether you are creating a new cluster or updating an existing one.

The SQL customer-managed resources (rpsql_service_account, rpsql_api_service_account, rpsql_cloud_storage_bucket, and rpsql_secret_manager_prefix) are immutable while SQL is enabled. To change any of them, you must first disable SQL.

Redpanda SQL deploys to a single availability zone (AZ), even when the cluster spans multiple AZs. In the rpsql block, set zones to one of the cluster’s zones. If you omit it, Redpanda uses the cluster’s first zone. The AZ is locked while Redpanda SQL is enabled. To move the SQL engine to a different AZ, disable and re-enable Redpanda SQL.

On a new cluster

  • Terraform

  • Cloud API

Creating a BYOVPC cluster with Redpanda SQL requires the Redpanda Terraform provider >= 2.1.1.

Add the SQL customer-managed resource fields and the rpsql block to the redpanda_cluster resource in your BYOVPC Terraform configuration, mapping the module outputs to the customer-managed resource fields. For a complete redpanda_cluster and redpanda_network example, see the GCP BYOVPC example in the Redpanda Terraform provider repository.

resource "redpanda_cluster" "cluster" {
  # ... existing BYOVPC cluster configuration ...
  customer_managed_resources = {
    gcp = {
      # ... existing BYOVPC customer-managed resources ...
      rpsql_service_account = {
        email = module.redpanda_gcp.redpanda_sql_service_account_email
      }
      rpsql_api_service_account = {
        email = module.redpanda_gcp.redpanda_sql_api_service_account_email
      }
      rpsql_cloud_storage_bucket = {
        name = module.redpanda_gcp.redpanda_sql_storage_bucket_name
      }
      rpsql_secret_manager_prefix = module.redpanda_gcp.redpanda_sql_secret_manager_prefix
    }
  }
  rpsql = {
    enabled  = true
    replicas = 1
    zones    = ["<sql-az>"]
  }
}

Replace the placeholders with your own values:

  • replicas: Initial number of SQL compute nodes (minimum 1, maximum 9).

  • <sql-az> (optional): The availability zone for the SQL engine.

Apply the configuration:

terraform apply

When creating a new BYOVPC cluster using the Cloud API, include the SQL customer-managed resource fields and the rpsql configuration in the POST /v1/clusters payload alongside your existing BYOVPC customer-managed resources.

  1. Authenticate to the Cloud API.

  2. Make a POST /v1/clusters request. The following payload shows only the SQL-specific fields. Include them alongside the full set of BYOVPC customer-managed resources your cluster requires, or the request fails. For the complete set, see the GCP BYOVPC example. Replace the placeholder values with your own:

    curl -X POST "https://api.redpanda.com/v1/clusters" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": {
      "name": "<cluster-name>",
      "type": "TYPE_BYOC",
      "cloud_provider": "CLOUD_PROVIDER_GCP",
      "region": "<region>",
      "zones": ["<zone-1>", "<zone-2>", "<zone-3>"],
      "network_id": "<network-id>",
      "throughput_tier": "<tier>",
      "resource_group_id": "<resource-group-id>",
      "customer_managed_resources": {
        "gcp": {
          "rpsql_service_account": {"email": "<rpsql-service-account-email>"},
          "rpsql_api_service_account": {"email": "<rpsql-api-service-account-email>"},
          "rpsql_cloud_storage_bucket": {"name": "<rpsql-storage-bucket-name>"},
          "rpsql_secret_manager_prefix": "<rpsql-secret-manager-prefix>"
        }
      },
      "rpsql": {
        "enabled": true,
        "replicas": <compute-nodes>,
        "zones": ["<sql-az>"]
      }
    }
  }'

    Replace the placeholders with your own values:

    • The four rpsql_* values: the service account emails, bucket name, and Secret Manager prefix from Provision SQL resources with Terraform.

    • <compute-nodes>: Initial number of SQL compute nodes (minimum 1, maximum 9).

    • <sql-az> (optional): The availability zone for the SQL engine.

  3. The request returns the ID of a long-running operation. Poll the GET /v1/operations/{operation.id} endpoint until the operation completes.

On an existing cluster

  • Terraform

  • Cloud Console

  • Cloud API

Enabling Redpanda SQL on an existing BYOVPC cluster with Terraform requires the Redpanda Terraform provider >= 2.2.0. On earlier versions, adding the SQL customer-managed resources replaces the cluster.

Add the SQL customer-managed resource fields and the rpsql block to the redpanda_cluster resource in your BYOVPC Terraform configuration, mapping the module outputs to the customer-managed resource fields:

resource "redpanda_cluster" "cluster" {
  # ... existing BYOVPC cluster configuration ...
  customer_managed_resources = {
    gcp = {
      # ... existing BYOVPC customer-managed resources ...
      rpsql_service_account = {
        email = module.redpanda_gcp.redpanda_sql_service_account_email
      }
      rpsql_api_service_account = {
        email = module.redpanda_gcp.redpanda_sql_api_service_account_email
      }
      rpsql_cloud_storage_bucket = {
        name = module.redpanda_gcp.redpanda_sql_storage_bucket_name
      }
      rpsql_secret_manager_prefix = module.redpanda_gcp.redpanda_sql_secret_manager_prefix
    }
  }
  rpsql = {
    enabled  = true
    replicas = 1
    zones    = ["<sql-az>"]
  }
}

Replace the placeholders with your own values:

  • replicas: Initial number of SQL compute nodes (minimum 1, maximum 9).

  • <sql-az> (optional): The availability zone for the SQL engine.

Apply the configuration:

terraform apply

  1. Log in to Redpanda Cloud and open your BYOVPC cluster.

  2. From the navigation menu, select Dataplane settings.

  3. On the Cluster tab, find the Redpanda SQL row and click Edit.

  4. In the dialog, enter the two service account emails, the GCS bucket name, and the Secret Manager prefix that you provisioned, then use the RPU slider to set the compute size.

  5. Click Enable Redpanda SQL engine.

The PATCH request must use an explicit leaf-path update_mask that lists each field you set, including each SQL customer-managed resource. Redpanda silently ignores a grouped mask (for example, customer_managed_resources), so the operation returns STATE_COMPLETED but SQL stays disabled.

  1. Authenticate to the Cloud API and obtain a bearer token.

  2. Make a PATCH /v1/clusters/{cluster.id} request. Set the update_mask query parameter to the explicit leaf paths, and provide the rpsql block and the four SQL customer-managed resources in the body. Replace {cluster.id} and the placeholder values with your own:

    curl -X PATCH "https://api.redpanda.com/v1/clusters/{cluster.id}?update_mask=rpsql.enabled,rpsql.replicas,rpsql.zones,customer_managed_resources.gcp.rpsql_service_account.email,customer_managed_resources.gcp.rpsql_api_service_account.email,customer_managed_resources.gcp.rpsql_cloud_storage_bucket.name,customer_managed_resources.gcp.rpsql_secret_manager_prefix" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "customer_managed_resources": {
      "gcp": {
        "rpsql_service_account": {"email": "<rpsql-service-account-email>"},
        "rpsql_api_service_account": {"email": "<rpsql-api-service-account-email>"},
        "rpsql_cloud_storage_bucket": {"name": "<rpsql-storage-bucket-name>"},
        "rpsql_secret_manager_prefix": "<rpsql-secret-manager-prefix>"
      }
    },
    "rpsql": {
      "enabled": true,
      "replicas": <compute-nodes>,
      "zones": ["<sql-az>"]
    }
  }'

    Replace the placeholders with your own values:

    • <compute-nodes>: Initial number of SQL compute nodes (minimum 1, maximum 9).

    • <sql-az> (optional): The availability zone for the SQL engine.

    • The four rpsql_* values: the service account emails, bucket name, and Secret Manager prefix from Provision SQL resources with Terraform.

      A successful enable request returns a long-running operation that stays STATE_IN_PROGRESS for roughly 15 to 30 minutes while the SQL node group provisions.

  3. Poll the GET /v1/operations/{operation.id} endpoint until the operation completes:

    curl -X GET "https://api.redpanda.com/v1/operations/{operation.id}" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json"

Verify the SQL engine is running

After you enable Redpanda SQL, a Redpanda SQL tile appears in the cluster overview. The tile shows the engine status, active sessions, queries per minute, and node count. When the Nodes indicator shows all nodes as ready (for example, 1/1), the engine is provisioned and accepting connections. Provisioning can take up to 30 minutes.

For the API flow, poll the long-running operation until it returns STATE_COMPLETED.

To verify the SQL engine is running, connect with psql version 16 or later, or another PostgreSQL client, using the connection details on the SQL tab under Connection details. Because a BYOVPC cluster is private, run the client from a host inside the cluster’s VPC. You can also query data directly using the SQL editor in the navigation menu.

BigLake access for Iceberg queries

To run Iceberg queries against topics backed by a Google BigLake catalog, the SQL engine needs BigLake access. On a BYOVPC cluster, the redpanda-data/redpanda-byovpc/gcp module grants this automatically when you enable Redpanda SQL (enable_redpanda_sql = true), so no additional IAM step is required. For the full BigLake catalog configuration, see Use Iceberg Topics with GCP Lakehouse.

Disable Redpanda SQL

Disabling Redpanda SQL tears down the SQL compute engine and clears its catalog state (catalog metadata, table mappings, and role/grant data). In-flight queries fail when SQL is disabled.

If you disable Redpanda SQL, Redpanda topic data, Schema Registry subjects, and any Iceberg-committed history for Iceberg-enabled topics are not affected. The Redpanda cluster itself continues to run normally, and only the SQL engine and its associated state are removed.

Re-enabling SQL on the same cluster provisions a fresh engine. Redpanda does not restore prior catalog state, table mappings, or grants. You must re-create catalogs, tables, and grants after re-enabling.

Disable SQL from the Cloud Console, or with the Terraform provider or the Cloud API.

  • Terraform

  • Cloud Console

  • Cloud API

Set rpsql.enabled to false in the redpanda_cluster resource, and remove the four SQL customer-managed resource fields (rpsql_service_account, rpsql_api_service_account, rpsql_cloud_storage_bucket, and rpsql_secret_manager_prefix) from customer_managed_resources.gcp. The control plane clears these resources when SQL is disabled, so leaving them in your configuration causes permanent drift.

resource "redpanda_cluster" "cluster" {
  # ... existing BYOVPC cluster configuration ...
  customer_managed_resources = {
    gcp = {
      # ... existing BYOVPC customer-managed resources ...
      # Remove the rpsql_* customer-managed resource fields.
    }
  }
  rpsql = {
    enabled = false
  }
}

Apply the change:

terraform apply

  1. Log in to Redpanda Cloud and open your cluster.

  2. From the navigation menu, select Dataplane settings.

  3. On the Cluster tab, find the Redpanda SQL row and click Edit.

  4. In the Edit Redpanda SQL engine dialog, click Disable.

Make a PATCH /v1/clusters/{cluster.id} request with rpsql.enabled set to false:

curl -X PATCH "https://api.redpanda.com/v1/clusters/{cluster.id}?update_mask=rpsql.enabled" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"rpsql": {"enabled": false}}'

Poll the GET /v1/operations/{operation.id} endpoint until the operation completes.

Disabling SQL does not delete the SQL-specific GCP resources (service accounts and bucket). These resources remain in your project, and you can reuse them if you re-enable SQL. To remove them, first wait until the disable operation completes, then set enable_redpanda_sql = false in the Terraform module and run terraform apply, or run terraform destroy to remove all BYOVPC resources. Removing these resources while the cluster still references them can cause the apply to fail.

Next steps