Redpanda Terraform Provider

The Redpanda Terraform provider allows you to manage your Redpanda Cloud infrastructure as code using Terraform. Terraform is an infrastructure-as-code tool that enables you to define, automate, and version-control your infrastructure configurations.

With the Redpanda Terraform provider, you can manage:

Why use Terraform with Redpanda?

  • Simplicity: Manage all your Redpanda Cloud resources in one place.

  • Automation: Create and modify resources without manual intervention.

  • Version Control: Track and roll back changes using version control systems, such as GitHub.

  • Scalability: Scale your infrastructure as your needs grow with minimal effort.

Understand Terraform configurations

Terraform configurations are written in HCL (HashiCorp Configuration Language), which is declarative. Here are the main building blocks of a Terraform configuration:

Providers

Providers tell Terraform how to communicate with the services you want to manage. For example, the Redpanda provider connects to the Redpanda Cloud API using client credentials.

provider "redpanda" {
  client_id     = "<your_client_id>"
  client_secret = "<your_client_secret>"
}

Resources

Resources define the infrastructure components you want to create, such as networks, clusters, or topics. Each resource block specifies the type of resource and its configuration.

resource "redpanda_network" "example" { (1)
  name           = "example-network" (2)
  cloud_provider = "aws" (3)
  region         = "us-east-1" (4)
  cidr_block     = "10.0.0.0/20" (5)
}
1 The resource type and internal name. The first part of this resource block specifies the type of resource being created. In this case, it is a redpanda_network, which defines a network for Redpanda Cloud. Different resource types include redpanda_cluster, redpanda_topic, and others. The second part is the internal name Terraform uses to identify this specific resource within your configuration. In this case, the internal name is example. This internal name allows you to reference the resource in other parts of your configuration. For example, redpanda_network.example.id can be used to access the unique ID of the network after it is created. The name does not affect the resource in Redpanda Cloud. It is for Terraform’s internal use.
2 A user-defined name for the resource as it will appear in Redpanda Cloud. This is the user-facing name visible in the Redpanda UI and API.
3 The cloud provider where the network is deployed, such as AWS or GCP.
4 The region where the resource will be provisioned.
5 The IP address range for the network.

Variables

Variables allow you to parameterize your configuration, making it reusable and customizable for different environments. Use variable blocks to define reusable values, like region, which can be overridden when running Terraform.

variable "region" {
  default = "us-east-1"
}

resource "redpanda_network" "example" {
  name           = "example-network"
  cloud_provider = "aws"
  region         = var.region
  cidr_block     = "10.0.0.0/20"
}

Outputs

Outputs let you extract information about your infrastructure, such as cluster URLs, to use in other configurations or scripts.

This example will display the cluster’s API URL after Terraform provisions the resources:

output "cluster_api_url" {
  value = data.redpanda_cluster.example.cluster_api_url
}

Limitations

The following functionality is supported in the Cloud API but not in the Redpanda Terraform provider:

  • Creating or deleting BYOVNet clusters on Azure

  • Secrets

  • Kafka Connect

  • Redpanda Connect

Do not modify throughput_tier after it is set. When allow_deletion is set to true, modifying throughput_tier forces replacement of the cluster: Terraform will destroy the existing cluster and create a new one, causing data loss.

Prerequisites

  1. Install at least version 1.0.0 of Terraform using the official guide.

  2. Create a service account in Redpanda Cloud:

    1. Log in to Redpanda Cloud.

    2. Navigate to the Organization IAM page and select the Service account tab. Click Create service account and provide a name for the new service account.

    3. Save the client ID and client secret for authentication.

Set up the provider

To set up the provider, you need to download the provider and authenticate to the Redpanda Cloud API. You can authenticate to the Redpanda Cloud API using environment variables or static credentials in your configuration file.

  1. Add the Redpanda provider to your Terraform configuration:

    terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = "~> 1.0"
    }
  }
}

  2. Initialize Terraform to download the provider:

    terraform init

  3. Add the credentials for the Redpanda Cloud service account you set in Prerequisites. In the Redpanda Cloud UI, find the client ID and client secret under Organization IAM → Service accounts. Set them as environment variables, or enter them in your Terraform configuration file:

    • Environment variables

    • Static credentials

    REDPANDA_CLIENT_ID=<client_id>
REDPANDA_CLIENT_SECRET=<client_secret>
    provider "redpanda" {
  client_id      = "<client_id>"
  client_secret  = "<client_secret>"
}

Examples

This section provides examples of using the Redpanda Terraform provider to create and manage clusters. For descriptions of resources and data sources, see the Redpanda Terraform Provider documentation.

For more information on the different cluster types mentioned in these examples, see Redpanda Cloud cluster types.

See the full list of zones and tiers available with each cloud provider in the Control Plane API reference.

Create a BYOC cluster

A BYOC (Bring Your Own Cloud) cluster allows you to provision a cluster in your own cloud account. This example creates a BYOC cluster on AWS with a custom network, resource group, and cluster configuration.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = "~> 1.0"
    }
  }
}

# Variables to parameterize the configuration
variable "resource_group_name" {
  description = "Name of the Redpanda resource group"
  default     = "testname"
}

variable "network_name" {
  description = "Name of the Redpanda network"
  default     = "testname"
}

variable "cluster_name" {
  description = "Name of the Redpanda BYOC cluster"
  default     = "test-cluster"
}

variable "region" {
  description = "Region for the Redpanda network and cluster"
  default     = "us-east-2"
}

variable "cloud_provider" {
  description = "Cloud provider for the Redpanda network"
  default     = "aws"
}

variable "zones" {
  description = "List of availability zones for the cluster"
  type        = list(string)
  default     = ["use2-az1", "use2-az2", "use2-az3"]
}

variable "cidr_block" {
  description = "CIDR block for the Redpanda network"
  default     = "10.0.0.0/20"
}

variable "throughput_tier" {
  description = "Throughput tier for the cluster"
  default     = "tier-1-aws-v2-x86"
}

# Redpanda provider configuration
provider "redpanda" {}

# Create a Redpanda resource group
resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

# Create a Redpanda network
resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"  # Specify BYOC cluster type
  cidr_block        = var.cidr_block
}

# Create a Redpanda BYOC cluster
resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "byoc"
  connection_type   = "public"  # Publicly accessible cluster
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true      # Allow the cluster to be deleted
  tags = {                      # Add metadata tags
    "environment" = "dev"
  }
}

Create a Dedicated cluster

A Dedicated cluster is fully managed by Redpanda and ensures consistent performance. This example provisions a cluster on AWS with specific zones and usage tiers.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = "~> 1.0"
    }
  }
}

# Variables for configuration
variable "resource_group_name" {
  description = "Name of the Redpanda resource group"
  default     = "test-dedicated-group"
}

variable "network_name" {
  description = "Name of the Redpanda network"
  default     = "dedicated-network"
}

variable "cluster_name" {
  description = "Name of the Redpanda dedicated cluster"
  default     = "dedicated-cluster"
}

variable "region" {
  description = "Region for the Redpanda network and cluster"
  default     = "us-west-1"
}

variable "cloud_provider" {
  description = "Cloud provider for the Redpanda network"
  default     = "aws"
}

variable "zones" {
  description = "List of availability zones for the cluster"
  type        = list(string)
  default     = ["usw1-az1", "usw1-az2", "usw1-az3"]
}

variable "cidr_block" {
  description = "CIDR block for the Redpanda network"
  default     = "10.1.0.0/20"
}

variable "throughput_tier" {
  description = "Throughput tier for the dedicated cluster"
  default     = "tier-1-aws-v2-arm"
}

# Redpanda provider configuration
provider "redpanda" {}

# Create a Redpanda resource group
resource "redpanda_resource_group" "test" {
  name = var.resource_group_name
}

# Create a Redpanda network
resource "redpanda_network" "test" {
  name              = var.network_name
  resource_group_id = redpanda_resource_group.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"  # Specify Dedicated cluster type
  cidr_block        = var.cidr_block
}

# Create a Redpanda dedicated cluster
resource "redpanda_cluster" "test" {
  name              = var.cluster_name
  resource_group_id = redpanda_resource_group.test.id
  network_id        = redpanda_network.test.id
  cloud_provider    = var.cloud_provider
  region            = var.region
  cluster_type      = "dedicated"
  connection_type   = "public"
  throughput_tier   = var.throughput_tier
  zones             = var.zones
  allow_deletion    = true
  aws_private_link = {  # Configure AWS PrivateLink for dedicated clusters
    enabled            = true
    connect_console    = true
    allowed_principals = ["arn:aws:iam::123456789024:root"]
  }
  tags = {
    "environment" = "dev"
  }
}

Create a Serverless cluster

A Serverless cluster is cost-effective and scales automatically based on usage. This example creates a cluster in the us-east-1 region with minimal configuration.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = "~> 1.0"
    }
  }
}

# Redpanda provider configuration
provider "redpanda" {}

# Define a resource group for the Serverless cluster
resource "redpanda_resource_group" "test" {
  name = var.resource_group_name  # Name of the resource group
}

# Create a Serverless cluster
resource "redpanda_serverless_cluster" "test" {
  name              = var.cluster_name                  # Name of the Serverless cluster
  resource_group_id = redpanda_resource_group.test.id   # Link to the resource group
  serverless_region = var.region                        # Specify the region for the cluster
}

# Variables for parameterizing the configuration
variable "resource_group_name" {
  description = "Name of the Redpanda resource group"
  default     = "testgroup"  # Default name for the resource group
}

variable "cluster_name" {
  description = "Name of the Redpanda Serverless cluster"
  default     = "testname"   # Default name for the Serverless cluster
}

variable "region" {
  description = "Region for the Serverless cluster"
  default     = "us-east-1"  # Default region for the cluster
}

Manage an existing cluster

To manage resources in existing Redpanda Cloud clusters, you must reference the cluster using the cluster ID (Redpanda ID). The following example creates a topic in a cluster with ID byoc-cluster-id. The redpanda_topic resource contains a field cluster_api_url that references the data.redpanda_cluster.byoc.cluster_api_url data resource.

data "redpanda_cluster" "byoc" {
  id = "byoc-cluster-id"
}

resource "redpanda_topic" "example" {
  name               = "example-topic"
  partition_count    = 3
  replication_factor = 3
  cluster_api_url    = data.redpanda_cluster.byoc.cluster_api_url
}

Manage Schema Registry and Schema Registry ACLs

You can also use Terraform to manage data plane resources, such as schemas and access controls, through the Redpanda Schema Registry.

The Redpanda Schema Registry provides centralized management of schemas for producers and consumers, ensuring compatibility and consistency of data serialized with formats such as Avro, Protobuf, or JSON Schema. Using the Redpanda Terraform provider, you can create, update, and delete schemas as well as manage fine-grained access control for Schema Registry resources.

You can use the following Terraform resources:

  • redpanda_schema: Defines and manages schemas in the Schema Registry.

  • redpanda_schema_registry_acl: Defines access control policies for Schema Registry subjects or registry-wide operations.

Create a schema

The redpanda_schema resource registers a schema in the Redpanda Schema Registry. Each schema is associated with a subject, which serves as the logical namespace for schema versioning. When you create or update a schema, Redpanda validates its compatibility level.

data "redpanda_cluster" "byoc" {
  id = "byoc-cluster-id"
}

resource "redpanda_user" "schema_user" {
  name            = "schema-user"
  password        = var.schema_password
  mechanism       = "scram-sha-256"
  cluster_api_url = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion  = true
}

resource "redpanda_schema" "user_events" {
  cluster_id  = data.redpanda_cluster.byoc.id
  subject     = "user_events-value"
  schema_type = "AVRO"

  schema = jsonencode({
    type = "record"
    name = "UserEvent"
    fields = [
      { name = "user_id", type = "string" },
      { name = "event_type", type = "string" },
      { name = "timestamp", type = "long" }
    ]
  })

  username = redpanda_user.schema_user.name
  password = var.schema_password
}

In this example:

  • cluster_id identifies the Redpanda cluster where the schema is stored.

  • subject defines the logical name under which schema versions are registered.

  • schema_type specifies the serialization type (AVRO, JSON, or PROTOBUF).

  • schema provides the full schema definition, encoded with jsonencode().

  • username and password authenticate the user to the Schema Registry.

Store credentials securely

Store credentials using environment variables or sensitive Terraform variables.

For short-lived credentials or CI/CD usage, use provider-level environment variables:

export REDPANDA_SR_USERNAME=schema-user
export REDPANDA_SR_PASSWORD="your-secret-password"

Or, declare a sensitive Terraform variable and inject it at runtime:

variable "schema_password" {
  description = "Password for the Schema Registry user"
  sensitive   = true
}

Then, set the value securely using an environment variable before running Terraform:

export TF_VAR_schema_password="your-secret-password"

This avoids committing secrets to source control.

Manage Schema Registry ACLs

The redpanda_schema_registry_acl resource configures fine-grained access control for Schema Registry subjects or registry-wide operations. Each ACL specifies which principal can perform specific operations on a subject or the registry.

resource "redpanda_schema_registry_acl" "allow_user_read" {
  cluster_id    = data.redpanda_cluster.byoc.id
  principal     = "User:${redpanda_user.schema_user.name}"
  resource_type = "SUBJECT"        # SUBJECT or REGISTRY
  resource_name = "user_events-value"
  pattern_type  = "LITERAL"        # LITERAL or PREFIXED
  host          = "*"
  operation     = "READ"           # READ, WRITE, DELETE, DESCRIBE, etc.
  permission    = "ALLOW"          # ALLOW or DENY
  username      = redpanda_user.schema_user.name
  password      = var.schema_password
}

In this example:

  • cluster_id identifies the cluster that hosts the Schema Registry.

  • principal specifies the user or service account (for example, User:alice).

  • resource_type determines whether the ACL applies to a specific SUBJECT or the entire REGISTRY.

  • resource_name defines the subject name (use * for wildcard).

  • pattern_type controls how the resource name is matched (LITERAL or PREFIXED).

  • operation defines the permitted action (READ, WRITE, DELETE, etc.).

  • permission defines whether the operation is allowed or denied.

  • host specifies the host filter (typically "*" for all hosts).

  • username and password authenticate the principal to the Schema Registry.

To manage Schema Registry ACLs, the user must have cluster-level ALTER permissions. This is typically granted through a Kafka ACL with ALTER on the CLUSTER resource.

Combine schema and ACLs

You can define both the schema and its ACLs in a single configuration to automate schema registration and access setup.

data "redpanda_cluster" "byoc" {
  id = "byoc-cluster-id"
}

resource "redpanda_user" "schema_user" {
  name            = "schema-user"
  password        = var.schema_password
  mechanism       = "scram-sha-256"
  cluster_api_url = data.redpanda_cluster.byoc.cluster_api_url
  allow_deletion  = true
}

resource "redpanda_schema" "user_events" {
  cluster_id  = data.redpanda_cluster.byoc.id
  subject     = "user_events-value"
  schema_type = "AVRO"

  schema = jsonencode({
    type = "record"
    name = "UserEvent"
    fields = [
      { name = "user_id", type = "string" },
      { name = "event_type", type = "string" },
      { name = "timestamp", type = "long" }
    ]
  })

  username = redpanda_user.schema_user.name
  password = var.schema_password
}

resource "redpanda_schema_registry_acl" "user_events_acl" {
  cluster_id    = data.redpanda_cluster.byoc.id
  principal     = "User:${redpanda_user.schema_user.name}"
  resource_type = "SUBJECT"
  resource_name = redpanda_schema.user_events.subject
  pattern_type  = "LITERAL"
  host          = "*"
  operation     = "READ"
  permission    = "ALLOW"
  username      = redpanda_user.schema_user.name
  password      = var.schema_password
}

This configuration registers an Avro schema for the user_events subject and grants a service account permission to read it from the Schema Registry.

Delete resources

Terraform provides a way to clean up your infrastructure when resources are no longer needed. The terraform destroy command deletes all the resources defined in your configuration.

Terraform ensures that dependent resources are deleted in the correct order. For example, a cluster dependent on a network will be removed after the network.

Delete all resources

  1. Navigate to the directory containing your Terraform configuration.

  2. Run the following command:

    terraform destroy

  3. Review the destruction plan Terraform generates. It will list all the resources to be deleted.

  4. Confirm by typing yes when prompted.

  5. Wait for the process to complete. Terraform will delete the resources and display a summary.

Delete specific resources

If you only want to delete a specific resource rather than everything in your configuration, use the -target flag with terraform destroy. For example:

terraform destroy -target=redpanda_network.example

This will delete only the redpanda_network.example resource.

Suggested reading