Cloud

Enable Redpanda SQL on a BYOVPC Cluster on AWS

BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or Redpanda Sales.

Enable Redpanda SQL on a Bring Your Own Virtual Private Cloud (BYOVPC) cluster on AWS so you can query streaming data in Redpanda topics using standard PostgreSQL syntax. You can enable SQL when creating a new BYOVPC cluster or on an existing one.

Unlike standard BYOC clusters, Redpanda does not create resources in your AWS account for BYOVPC clusters. You must provision the SQL-specific AWS resources yourself and supply them to the cluster as customer-managed resources (CMRs) before enabling the SQL engine. You can enable SQL on an existing cluster from the Cloud Console, the Terraform provider, or the Cloud API. To enable SQL when you create a new cluster, use the Terraform provider or the Cloud API.

After reading this page, you will be able to:

  • Provision SQL-specific AWS resources using the Redpanda BYOVPC Terraform module

  • Enable Redpanda SQL on a new or existing BYOVPC cluster by supplying customer-managed resources

  • Verify that the SQL engine is running and ready to accept connections

Prerequisites

  • For a new cluster: a BYOVPC network on AWS. For an existing cluster: a deployed BYOVPC cluster on AWS.

  • The Redpanda BYOVPC Terraform module and Terraform version 1.8.5 or later, to provision the SQL resources.

  • If you enable SQL with Terraform: the Redpanda Terraform provider, configured with valid credentials. Enabling SQL on a new cluster requires version 2.1.0 or later; enabling on an existing cluster requires version 2.2.0 or later, because earlier versions replace the cluster when you add the SQL customer-managed resources.

  • Admin permissions in your Redpanda Cloud organization on usage-based billing. If using the Cloud API, a valid bearer token with permission to update clusters. See Authenticate to the Cloud API.

  • To verify the engine after enabling it: psql version 16 or later on a host inside the cluster’s VPC. A BYOVPC cluster is private, so the SQL endpoint is not reachable from outside the VPC.

Provision SQL resources with Terraform

In your redpanda-data/redpanda-byovpc/aws module configuration, set enable_redpanda_sql = true:

module "redpanda_byovpc" {
  source = "redpanda-data/redpanda-byovpc/aws"
  # ... existing configuration ...
  enable_redpanda_sql = true
}

Apply the updated configuration:

terraform apply

This creates three SQL-specific resources in your AWS account:

  • An IAM role and instance profile for SQL compute nodes

  • An S3 bucket for SQL data storage (versioning disabled)

  • A security group for SQL nodes

After applying, Terraform outputs the following values. Supply these ARNs to the cluster when you enable SQL.

Module output Description

rpsql_node_group_instance_profile_arn

ARN of the IAM instance profile for SQL nodes

rpsql_cloud_storage_bucket_arn

ARN of the S3 bucket for SQL data storage

rpsql_security_group_arn

ARN of the security group for SQL nodes

If you plan to query Iceberg topics backed by AWS Glue, this module can also provide the Glue catalog policy that the SQL engine needs. See Grant Glue access for Iceberg queries.

Enable Redpanda SQL

Use the Terraform provider, Cloud API, or Cloud Console to enable the SQL engine. The steps differ depending on whether you are creating a new cluster or updating an existing one.

The SQL customer-managed resources (rpsql_node_group_instance_profile, rpsql_cloud_storage_bucket, rpsql_security_group) are immutable while SQL is enabled. To change any of these resources, you must first disable SQL.

Redpanda SQL deploys to a single availability zone (AZ), even when the cluster spans multiple AZs. In the rpsql block, set zones to one of the cluster’s zones. If you omit it, Redpanda uses the cluster’s first zone. The AZ is locked while Redpanda SQL is enabled. To move the SQL engine to a different AZ, disable and re-enable Redpanda SQL.

On a new cluster

  • Terraform

  • Cloud API

Creating a BYOVPC cluster with Redpanda SQL requires the Redpanda Terraform provider >= 2.1.0.

Add the SQL customer-managed resource fields and the rpsql block to the redpanda_cluster resource in your BYOVPC Terraform configuration. For the base redpanda_cluster configuration, see Create a BYOVPC Cluster on AWS.

resource "redpanda_cluster" "cluster" {
  # ... existing BYOVPC cluster configuration ...
  customer_managed_resources = {
    aws = {
      # ... existing BYOVPC customer-managed resources ...
      rpsql_node_group_instance_profile = {
        arn = module.redpanda_byovpc.rpsql_node_group_instance_profile_arn
      }
      rpsql_cloud_storage_bucket = {
        arn = module.redpanda_byovpc.rpsql_cloud_storage_bucket_arn
      }
      rpsql_security_group = {
        arn = module.redpanda_byovpc.rpsql_security_group_arn
      }
    }
  }
  rpsql = {
    enabled  = true
    replicas = 1
    zones    = ["<sql-az>"]
  }
}

Replace the placeholders with your own values:

  • replicas: Initial number of SQL compute nodes (minimum 1, maximum 9).

  • <sql-az> (optional): The availability zone for the SQL engine.

Apply the configuration:

terraform apply

When creating a new BYOVPC cluster using the Cloud API, include the SQL customer-managed resource ARNs and the rpsql configuration in the POST /v1/clusters payload alongside your existing BYOVPC customer-managed resources.

  1. Authenticate to the Cloud API.

  2. Make a POST /v1/clusters request. Include your existing BYOVPC customer-managed resource fields (see Create a BYOVPC Cluster on AWS) and add the SQL customer-managed resource fields and rpsql block:

    curl -X POST "https://api.redpanda.com/v1/clusters" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": {
      "name": "<cluster-name>",
      "type": "TYPE_BYOC",
      "cloud_provider": "CLOUD_PROVIDER_AWS",
      "region": "<region>",
      "zones": ["<az-1>", "<az-2>", "<az-3>"],
      "network_id": "<network-id>",
      "throughput_tier": "<tier>",
      "resource_group_id": "<resource-group-id>",
      "customer_managed_resources": {
        "aws": {
          "rpsql_node_group_instance_profile": {"arn": "<rpsql_node_group_instance_profile_arn>"},
          "rpsql_cloud_storage_bucket": {"arn": "<rpsql_cloud_storage_bucket_arn>"},
          "rpsql_security_group": {"arn": "<rpsql_security_group_arn>"}
        }
      },
      "rpsql": {
        "enabled": true,
        "replicas": <compute-nodes>,
        "zones": ["<sql-az>"]
      }
    }
  }'

    Replace the placeholders with your own values:

    • <rpsql_node_group_instance_profile_arn>, <rpsql_cloud_storage_bucket_arn>, <rpsql_security_group_arn>: The corresponding outputs from the Terraform module.

    • <compute-nodes>: Initial number of SQL compute nodes (minimum 1, maximum 9).

    • <sql-az> (optional): The availability zone for the SQL engine.

  3. The request returns the ID of a long-running operation. Poll the GET /v1/operations/{operation.id} endpoint until the operation completes:

    curl -X GET "https://api.redpanda.com/v1/operations/{operation.id}" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json"

    When the operation is complete, the response shows "state": "STATE_COMPLETED".

On an existing cluster

  • Terraform

  • Cloud Console

  • Cloud API

Enabling Redpanda SQL on an existing BYOVPC cluster with Terraform requires the Redpanda Terraform provider >= 2.2.0. On earlier versions, adding the SQL customer-managed resources replaces the cluster.

Add the SQL customer-managed resource fields and the rpsql block to the redpanda_cluster resource in your BYOVPC Terraform configuration:

resource "redpanda_cluster" "cluster" {
  # ... existing BYOVPC cluster configuration ...
  customer_managed_resources = {
    aws = {
      # ... existing BYOVPC customer-managed resources ...
      rpsql_node_group_instance_profile = {
        arn = module.redpanda_byovpc.rpsql_node_group_instance_profile_arn
      }
      rpsql_cloud_storage_bucket = {
        arn = module.redpanda_byovpc.rpsql_cloud_storage_bucket_arn
      }
      rpsql_security_group = {
        arn = module.redpanda_byovpc.rpsql_security_group_arn
      }
    }
  }
  rpsql = {
    enabled  = true
    replicas = 1
    zones    = ["<sql-az>"]
  }
}

Replace the placeholders with your own values:

  • replicas: Initial number of SQL compute nodes (minimum 1, maximum 9).

  • <sql-az> (optional): The availability zone for the SQL engine.

Apply the configuration:

terraform apply

  1. Log in to Redpanda Cloud and open your BYOVPC cluster.

  2. From the navigation menu, select Dataplane settings.

  3. On the Cluster tab, find the Redpanda SQL row and click Edit.

  4. In the dialog, enter the ARNs from the Terraform module outputs, then use the RPU slider to set the compute size.

  5. Click Enable Redpanda SQL engine.

The PATCH request must use an explicit leaf-path update_mask that lists each field you set, including each SQL customer-managed resource. Redpanda silently ignores a grouped mask (for example, customer_managed_resources), so the operation returns STATE_COMPLETED but SQL stays disabled.

  1. Authenticate to the Cloud API.

  2. Locate the cluster ID in the Details section of the cluster overview in the Cloud Console.

  3. Make a PATCH /v1/clusters/{cluster.id} request. Set the update_mask query parameter to the explicit leaf paths, and provide the rpsql block and the SQL customer-managed resources in the body. Replace {cluster.id} with your cluster ID:

    curl -X PATCH "https://api.redpanda.com/v1/clusters/{cluster.id}?update_mask=rpsql.enabled,rpsql.replicas,rpsql.zones,customer_managed_resources.aws.rpsql_node_group_instance_profile.arn,customer_managed_resources.aws.rpsql_security_group.arn,customer_managed_resources.aws.rpsql_cloud_storage_bucket.arn" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "customer_managed_resources": {
      "aws": {
        "rpsql_node_group_instance_profile": {"arn": "<rpsql_node_group_instance_profile_arn>"},
        "rpsql_cloud_storage_bucket": {"arn": "<rpsql_cloud_storage_bucket_arn>"},
        "rpsql_security_group": {"arn": "<rpsql_security_group_arn>"}
      }
    },
    "rpsql": {
      "enabled": true,
      "replicas": <compute-nodes>,
      "zones": ["<sql-az>"]
    }
  }'

    Replace the placeholders with your own values:

    • <rpsql_node_group_instance_profile_arn>, <rpsql_cloud_storage_bucket_arn>, <rpsql_security_group_arn>: The corresponding outputs from the Terraform module.

    • <compute-nodes>: Initial number of SQL compute nodes (minimum 1, maximum 9).

    • <sql-az> (optional): The availability zone for the SQL engine.

  4. The request returns the ID of a long-running operation. Poll the GET /v1/operations/{operation.id} endpoint until the operation completes:

    curl -X GET "https://api.redpanda.com/v1/operations/{operation.id}" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json"

    When the operation is complete, the response shows "state": "STATE_COMPLETED".

Verify the SQL engine is running

After you enable Redpanda SQL, a Redpanda SQL tile appears in the cluster overview. The tile shows the engine status, active sessions, queries per minute, and node count. When the Nodes indicator shows all nodes as ready (for example, 1/1), the engine is provisioned and accepting connections. Provisioning can take up to 30 minutes.

For the API flow, poll the long-running operation until it returns STATE_COMPLETED.

To verify the SQL engine is running, connect with psql version 16 or later, or another PostgreSQL client, using the connection details on the SQL tab under Connection details. Because a BYOVPC cluster is private, run the client from a host inside the cluster’s VPC. You can also query data directly using the SQL editor in the navigation menu.

Grant Glue access for Iceberg queries

To run Iceberg queries against topics backed by an AWS Glue catalog, the SQL engine needs Glue permissions. On a BYOVPC cluster, the redpanda-data/redpanda-byovpc/aws module can’t attach these automatically, because the SQL engine’s IAM role doesn’t exist until the cluster is created. After the cluster is running:

  1. Set enable_glue_iceberg_catalog = true on the redpanda_byovpc module. This widens the agent permissions boundary and exports a Glue policy as glue_iceberg_policy_arn.

  2. Attach that policy to both of the following roles from the workspace that creates the cluster:

    • redpanda-cloud-storage-manager-<cluster-id>: the broker role that writes topic data to the Glue catalog. Without this attachment, nothing is written to Glue, so there is nothing to query.

    • redpanda-<cluster-id>-redpanda-oxla-cluster: the SQL engine role that reads the catalog to run queries.

      resource "aws_iam_role_policy_attachment" "glue_storage_manager" {
  role       = "redpanda-cloud-storage-manager-${redpanda_cluster.cluster.id}"
  policy_arn = module.redpanda_byovpc.glue_iceberg_policy_arn
}

resource "aws_iam_role_policy_attachment" "glue_rpsql_engine" {
  role       = "redpanda-${redpanda_cluster.cluster.id}-redpanda-oxla-cluster"
  policy_arn = module.redpanda_byovpc.glue_iceberg_policy_arn
}

Without both attachments, REFRESH of the Glue-backed catalog fails with a permissions error. For the full AWS Glue catalog configuration, see Query Iceberg Topics using AWS Glue.

Disable Redpanda SQL

Disabling Redpanda SQL tears down the SQL compute engine and clears its catalog state (catalog metadata, table mappings, and role/grant data). In-flight queries fail when SQL is disabled.

If you disable Redpanda SQL, Redpanda topic data, Schema Registry subjects, and any Iceberg-committed history for Iceberg-enabled topics are not affected. The Redpanda cluster itself continues to run normally, and only the SQL engine and its associated state are removed.

Re-enabling SQL on the same cluster provisions a fresh engine. Redpanda does not restore prior catalog state, table mappings, or grants. You must re-create catalogs, tables, and grants after re-enabling.

Disable SQL from the Cloud Console, or with the Terraform provider or the Cloud API.

  • Terraform

  • Cloud Console

  • Cloud API

Set rpsql.enabled to false in the redpanda_cluster resource, and remove the three SQL customer-managed resource fields (rpsql_node_group_instance_profile, rpsql_cloud_storage_bucket, and rpsql_security_group) from customer_managed_resources.aws. The control plane clears these resources when SQL is disabled, so leaving them in your configuration causes permanent drift.

resource "redpanda_cluster" "cluster" {
  # ... existing BYOVPC cluster configuration ...
  customer_managed_resources = {
    aws = {
      # ... existing BYOVPC customer-managed resources ...
      # Remove the rpsql_* customer-managed resource fields.
    }
  }
  rpsql = {
    enabled = false
  }
}

Apply the change:

terraform apply

  1. Log in to Redpanda Cloud and open your cluster.

  2. From the navigation menu, select Dataplane settings.

  3. On the Cluster tab, find the Redpanda SQL row and click Edit.

  4. In the Edit Redpanda SQL engine dialog, click Disable.

Make a PATCH /v1/clusters/{cluster.id} request with rpsql.enabled set to false, using an explicit update_mask. Replace {cluster.id} with your cluster ID:

curl -X PATCH "https://api.redpanda.com/v1/clusters/{cluster.id}?update_mask=rpsql.enabled" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"rpsql":{"enabled":false}}'

The request returns the ID of a long-running operation. Poll GET /v1/operations/{operation.id} until the operation completes.

Disabling SQL does not delete the SQL-specific AWS resources (IAM instance profile, S3 bucket, and security group); they remain in your account, and you can reuse them if you re-enable SQL. To remove them, first wait until the disable operation completes, then set enable_redpanda_sql = false in the Terraform module and run terraform apply, or run terraform destroy to remove all BYOVPC resources. Removing these resources while the cluster still references them can cause the apply to fail.

Next steps