Redpanda Cloud Overview

Redpanda Cloud is a complete data streaming platform delivered as a fully-managed service with automated upgrades and patching, data and partition balancing, and 24x7 support. It continuously monitors and maintains your clusters along with the underlying infrastructure to meet strict performance, availability, reliability, and security requirements. All Redpanda Cloud clusters are deployed with an integrated Redpanda Console.

For more detailed information about the Redpanda platform, see Introduction to Redpanda and How Redpanda Works.

Cluster types

Redpanda offers three types of fully-managed cloud clusters:

  • Serverless: Clusters hosted in Redpanda Cloud. This is the fastest and easiest way to start data streaming.

  • Dedicated Cloud: Single-tenant clusters hosted in Redpanda Cloud. This provides more control over your deployment.

  • Bring Your Own Cloud (BYOC): Clusters hosted in your private cloud. This provides more control over your deployment and offers full data sovereignty.

    With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including service accounts, subnetworks, IAM roles, firewall rules, and storage buckets. For the most security, you can manage these resources yourself with a customer-managed VPC.

Serverless

With Serverless clusters, you host your data in Redpanda’s VPC, and Redpanda handles automatic scaling, provisioning, operations, and maintenance. This is a production-ready deployment option with a cluster available instantly. There is no base cost, and with pay-as-you-go billing after the free trial, you only pay for what you consume. You can view detailed billing activity for each cluster and edit payment methods on the Billing page.

Sign up for Serverless

To start using Serverless, sign up for a free trial. New trials receive $100 (USD) in free credits to spend in the first 14 days. This should be enough to run Redpanda with reasonable throughput. No credit card is required for a trial. To continue using your Serverless cluster after the free trial, add a credit card and pay as you go.

You can also subscribe to Redpanda Cloud through AWS Marketplace to quickly provision Serverless clusters. With a usage-based pay-as-you-go subscription, you only pay for what you use and can cancel anytime.

Serverless is currently in a limited availability (LA) release with usage limits. During LA, existing clusters can scale to the usage limits, but new clusters may need to wait for availability.

Dedicated Cloud

With Dedicated clusters, you host your data on Redpanda cloud resources (AWS, GCP, or Azure), and Redpanda handles provisioning, operations, and maintenance. Dedicated clusters are single-tenant deployments that support private networking (for example, VPC peering to talk over private IPs) for better data isolation. When you create a Dedicated cluster, you select the supported tier that meets your compute and storage needs.

Sign up for Dedicated

Subscribe to Redpanda Cloud on the AWS Marketplace to quickly provision Dedicated clusters. New subscriptions receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime.

Alternatively, you can contact Redpanda sales to request a private offer for monthly or annual committed use. With a usage-based billing commitment, you sign up for a monthly or an annual minimum spend amount. You can then use Google Cloud Marketplace or AWS Marketplace to quickly provision Dedicated clusters, and you can view invoices and manage your subscription in the marketplace.

Bring Your Own Cloud (BYOC)

With BYOC clusters, you deploy Redpanda in your own cloud (AWS, GCP, or Azure), and all data is contained in your own environment. This provides an additional layer of security and isolation. When you create a BYOC cluster, you select the supported tier that meets your compute and storage needs. Redpanda handles provisioning, operations, and maintenance. See also: BYOC architecture.

Sign up for BYOC

To start using BYOC, contact Redpanda sales to request a private offer. You are billed directly or through Google Cloud Marketplace or AWS Marketplace.

Serverless vs Dedicated/BYOC

Serverless clusters are a good fit for the following:

  • Starter and growing workloads

  • Spiky workloads (that is, development environments, systems that only occasionally get busy, or workloads that come and go)

  • Fast and dynamic cluster creation: you can use a Serverless cluster as an isolated container for topics

With Serverless (or if you sign up for Dedicated through the AWS Marketplace), you only pay for what you consume, without any commitment. A cluster is created instantly, so you can surface it in your applications (for example, for tenant isolation). If your workload increases, you can migrate it to a Dedicated or BYOC cluster.

Dedicated and BYOC clusters offer the following features:

  • Single-zone or multi-zone availability (a multi-zone cluster provides higher resiliency in the event of a failure in one of the zones)

  • Private networking using VPC peering or a private connectivity service, such as AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.

  • Ability to export metrics to a 3rd-party monitoring system

  • Managed connectors

  • Higher limits and quotas (see Serverless limits)

Consider Dedicated or BYOC if you need any of the following:

  • Control over the deployment

  • Private networking

  • Workloads with consistently-high throughput

Shared responsibility model

The Redpanda Cloud shared responsibility model lists the security ownership areas for Redpanda and customers. Responsibilities depend on the type of deployment.

  • Dedicated

  • BYOC

  • BYOC with customer-managed VPC

Resource Redpanda responsibility Customer responsibility

Redpanda upgrades and hotfixes

Cost management and attribution

Software vulnerability remediation

Infrastructure vulnerability remediation

IAM (roles, service accounts, access segmentation)

Compute

VPC (subnets, routing, firewall)

VPC peering

VPC private links (service endpoint)

VPC private links (consumer endpoint)

Local storage

Tiered Storage

Control plane

Access controls and audit

Managed disaster recovery

Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks)

Availability SLA

Proactive threat detection

Static secret rotation

Incident response

Resilience verification

Kafka Connect infrastructure

Kafka Connect tasks state

Resource Redpanda responsibility Customer responsibility

Redpanda upgrades and hotfixes

Cost management and attribution

Software vulnerability remediation

Infrastructure vulnerability remediation

IAM (roles, service accounts, access segmentation)

Compute

VPC (subnets, routing, firewall)

VPC peering

VPC private links (service endpoint)

VPC private links (consumer endpoint)

Local storage

Tiered Storage

Control plane

Access controls and audit

Managed disaster recovery

Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks)

Availability SLA

✓ (subject to required access to customer resources)

Proactive threat detection

Static secret rotation

Incident response

Resilience verification

Kafka Connect infrastructure

Kafka Connect tasks state

Resource Redpanda responsibility Customer responsibility

Redpanda upgrades and hotfixes

Cost management and attribution

Software vulnerability remediation

Infrastructure vulnerability remediation

IAM (roles, service accounts, access segmentation)

Compute

VPC (subnets, routing, firewall)

VPC peering

VPC private links (service endpoint)

VPC private links (consumer endpoint)

Local storage

Tiered Storage

Control plane

Access controls and audit

Managed disaster recovery

Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks)

✓ (for VPC components and cloud storage buckets/containers managed by customer)

Availability SLA

✓ (subject to required access to customer resources)

Proactive threat detection

Static secret rotation

Incident response

Resilience verification

Kafka Connect infrastructure

Kafka Connect tasks state

Maintenance windows

Redpanda runs maintenance and upgrade operations on clusters in a rolling fashion, accompanied by a series of health checks, so there is no disruption to the availability of your service. As part of the Kafka protocol, recycling nodes cause client connections to be restarted. All mainstream client libraries support automatic reconnections for this.

By default, Redpanda Cloud may run maintenance operations on any day at any time. You can override this default and schedule a specific maintenance window on the Cluster settings page. A Scheduled maintenance window requires Redpanda Cloud to run operations on the day and time specified. Maintenance windows typically take six hours. All operations begin during the maintenance window, but some operations may complete after the window ends. All times are in Coordinated Universal Time (UTC).

Redpanda Cloud maintenance cycles always start on Tuesdays. Clusters scheduled on Tuesdays are updated first, and clusters scheduled on Mondays are updated last. Keep this in my mind when sequencing updates for multiple clusters.

Redpanda Cloud architecture

When you sign up for a Redpanda account, Redpanda creates an organization for you. Your organization contains all your Redpanda resources, including your clusters and networks. Within your organization, Redpanda creates a default resource group to contain your resources. You can rename this resource group, and you can create more resource groups. For example, you may want different resource groups for production and testing.

For high availability, Redpanda Cloud uses a control plane and data plane architecture.

  • Control plane: This is where most cluster management, operations, and maintenance takes place. The control plane enforces rules in the data plane.

  • Data plane: This is where your cluster lives. The term data plane is used interchangeably with cluster.

  • Agent: Redpanda uses an agent to manage the data plane from the control plane.

Clusters are configured and maintained in the control plane, but they remain available even if the network connection to the control plane is lost.

In the user interface, when you’re at the organization (org) level or the resource group level, but you haven’t yet selected a cluster, you’re in the control plane. This is where you can select, create, and delete clusters, resource groups, and networks. When you’re at the cluster level working with topics, consumer groups, and connectors, you’re in the data plane.

BYOC architecture

The following diagram shows a BYOC architecture, where you deploy the data plane in your own VPC. All network connections into the data plane take place through either a public endpoint or a VPC peering network connection. Sensitive data and credentials never leave the data plane.

Data plane and control plane

A BYOC cluster is initially set up from the control plane. This is a two-step process performed by rpk cloud byoc apply:

  1. You bootstrap a virtual machine (VM) in your VPC.

    This VM spins up the agent and the required infrastructure. Redpanda assigns the necessary IAM policies required to run the agent and configures workload identity. That is, it configures independent IAM roles for each workload, with only the permissions each workload requires.

  2. The agent communicates with the control plane to pull the cluster specifications.

    After the agent is up and running, it connects to the control plane and starts dequeuing and applying cluster specifications that provision, configure, and maintain clusters. The agent is in constant communication with the control plane, receiving and applying cluster specifications and exchanging cluster metadata. Agents are authenticated and authorized through opaque and ephemeral tokens, and they have dedicated job queues in the control plane.

    Agents also manage VPC peering networks. One agent manages each network resource at a time.

    cloud_byoc_apply
To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run rpk cloud byoc apply with your cloud provider.

Redpanda Cloud vs Self-Managed feature compatibility

Redpanda Cloud deployments do not support the following functionality available in Redpanda Self-Managed deployments:

  • FIPS-compliance mode

  • Data transforms (currently in beta for Redpanda Cloud)

  • Remote Read Replicas (currently in beta for Redpanda Cloud)

  • Kafka API OIDC authentication (however, Redpanda Cloud does support user authentication to an organization with SSO)

  • Kerberos authentication

  • Redpanda Console topic documentation

  • Setting auto_create_topics_enabled=true

  • Admin API

  • The following rpk commands:

    • rpk connect commands

    • rpk redpanda commands

    • rpk cluster config

    • rpk cluster health

    • rpk cluster license

    • rpk cluster maintenance

    • rpk cluster partitions

    • rpk cluster self-test

    • rpk cluster storage

    • rpk generate app (supported in Serverless clusters)

    • rpk security user (supported in Serverless clusters)

    • rpk topic describe-storage (all other rpk topic commands are supported on both Redpanda Cloud and self-hosted)

    • rpk transform commands (currently in beta for Redpanda Cloud)

The rpk cloud commands are not supported in self-hosted deployments.