Redpanda Cloud Overview
Redpanda Cloud is a complete data streaming platform delivered as a fully-managed service with automated upgrades and patching, data and partition balancing, and 24x7 support. It continuously monitors and maintains your clusters along with the underlying infrastructure to meet strict performance, availability, reliability, and security requirements. All Redpanda Cloud clusters are deployed with an integrated Redpanda Console.
Cluster types
Redpanda offers three types of fully-managed cloud clusters:
-
Serverless: Clusters hosted in Redpanda Cloud. This is the fastest and easiest way to start data streaming.
-
Dedicated Cloud: Single-tenant clusters hosted in Redpanda Cloud. This provides more control over your deployment.
-
Bring Your Own Cloud (BYOC): Clusters hosted in your private cloud. This provides more control over your deployment and offers full data sovereignty.
With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including service accounts, subnetworks, IAM roles, firewall rules, and storage buckets. For the most security, you can manage these resources yourself with a customer-managed VPC.
Serverless
With Serverless clusters, you host your data in Redpanda’s VPC, and Redpanda handles automatic scaling, provisioning, operations, and maintenance. This is a production-ready deployment option with a cluster available instantly. This is no base cost, and with pay-as-you-go billing after the free trial, you only pay for what you consume.
Available on AWS.
Sign up for Serverless
To start using Serverless, sign up for a free trial. New trials receive $100 (USD) in free credits to spend in the first 14 days. This should be enough to run Redpanda with reasonable throughput. No credit card is required for a trial. To continue using Serverless, add a credit card and pay as you go. You can edit payment methods on the Redpanda Cloud Billing page.
| Serverless is currently in a limited availability (LA) release with usage limits and no public SLA. During LA, existing clusters can scale to the usage limits, but new clusters may need to wait for availability. |
Dedicated Cloud
With Dedicated clusters, you host your data in Redpanda’s VPC, and Redpanda handles provisioning, operations, and maintenance. Dedicated clusters are single-tenant deployments that support private networking (that is, VPC peering to talk over private IPs) for better data isolation.
Available on AWS or GCP.
Sign up for Dedicated
To start using Dedicated, contact Redpanda sales to request a private offer for monthly or annual committed use. With a usage-based billing commitment, you sign up for a monthly or an annual minimum spend amount. You can then use Google Cloud Marketplace or AWS Marketplace to quickly provision Dedicated Cloud clusters, and you can view invoices and manage your subscription in the marketplace.
Bring Your Own Cloud (BYOC)
With BYOC clusters, you deploy Redpanda in your own VPC, so all data is contained in your own environment. This provides an additional layer of security and isolation. Redpanda handles provisioning, operations, and maintenance. See BYOC architecture.
Available on AWS or GCP.
Sign up for BYOC
To start using BYOC, contact Redpanda sales to request a private offer, and you can be billed either directly or through Google Cloud Marketplace or AWS Marketplace.
Serverless vs Dedicated/BYOC
Serverless clusters are a good fit for the following:
-
Starter and growing workloads
-
Spiky workloads (that is, development environments, systems that only occasionally get busy, or workloads that come and go)
-
Fast and dynamic cluster creation: you can use a Serverless cluster as an isolated container for topics
With Serverless, you only pay for what you consume, without any commitment. A cluster is created instantly, so you can surface it in your applications (for example, for tenant isolation). If your workload increases, you can migrate it to a Dedicated or BYOC cluster.
Dedicated and BYOC clusters offer the following features:
-
Single-zone or multi-zone availability (a multi-zone cluster provides higher resiliency in the event of a failure in one of the zones)
-
Private networking using VPC peering
-
Ability to export metrics to a 3rd-party monitoring system
-
Managed connectors
-
Higher limits and quotas (see Serverless limits)
Consider Dedicated or BYOC if you need any of the following:
-
Control over the deployment
-
Private networking
-
Workloads with consistently-high throughput
Cluster tiers
When you create a Dedicated or BYOC cluster, you select a throughput tier. The following table lists the current cluster-level AWS and GCP tier limits. Legacy tiers may have different limits. For more information, contact support.
| Tier | Maximum ingress | Maximum egress | Maximum logical partitions | Maximum connections |
|---|---|---|---|---|
Tier 1 |
20 MBps |
60 MBps |
1,000 |
1,500 |
Tier 2 |
50 MBps |
150 MBps |
2,800 |
3,750 |
Tier 3 |
100 MBps |
200 MBps |
5,600 |
7,500 |
Tier 4 |
200 MBps |
400 MBps |
11,300 |
15,000 |
Tier 5 |
400 MBps |
800 MBps |
22,800 |
30,000 |
Tier 6 (BYOC only) |
800 MBps |
1,600 MBps |
45,000 |
60,000 |
Tier 7 (BYOC only) |
1,200 MBps |
2,400 MBps |
67,500 |
90,000 |
| Serverless clusters are not on tiers. See Serverless limits. To change your cluster type, contact support. |
Shared responsibility model
The Redpanda Cloud shared responsibility model lists the security ownership areas for Redpanda and customers. Responsibilities depend on the type of deployment.
-
Dedicated
-
BYOC
-
BYOC with customer-managed VPC
| Resource | Redpanda responsibility | Customer responsibility |
|---|---|---|
Redpanda upgrades and hotfixes |
✓ |
|
Cost management and attribution |
✓ |
|
Software vulnerability remediation |
✓ |
|
Infrastructure vulnerability remediation |
✓ |
|
IAM (roles, service accounts, access segmentation) |
✓ |
|
Compute |
✓ |
|
VPC (subnets, routing, firewall) |
✓ |
|
VPC peering |
✓ |
|
VPC private links (service endpoint) |
✓ |
|
VPC private links (consumer endpoint) |
✓ |
|
Local storage |
✓ |
|
Tiered Storage |
✓ |
|
Control plane |
✓ |
|
Access controls and audit |
✓ |
|
Managed disaster recovery |
✓ |
|
Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks) |
✓ |
|
Availability SLA |
✓ |
|
Proactive threat detection |
✓ |
|
Static secret rotation |
✓ |
|
Incident response |
✓ |
|
Resilience verification |
✓ |
| Resource | Redpanda responsibility | Customer responsibility |
|---|---|---|
Redpanda upgrades and hotfixes |
✓ |
|
Cost management and attribution |
✓ |
✓ |
Software vulnerability remediation |
✓ |
|
Infrastructure vulnerability remediation |
✓ |
|
IAM (roles, service accounts, access segmentation) |
✓ |
✓ |
Compute |
✓ |
|
VPC (subnets, routing, firewall) |
✓ |
✓ |
VPC peering |
✓ |
|
VPC private links (service endpoint) |
✓ |
|
VPC private links (consumer endpoint) |
✓ |
|
Local storage |
✓ |
|
Tiered Storage |
✓ |
|
Control plane |
✓ |
|
Access controls and audit |
✓ |
✓ |
Managed disaster recovery |
✓ |
|
Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks) |
✓ |
|
Availability SLA |
✓ (subject to required access to customer resources) |
|
Proactive threat detection |
✓ |
✓ |
Static secret rotation |
✓ |
|
Incident response |
✓ |
|
Resilience verification |
✓ |
| Resource | Redpanda responsibility | Customer responsibility |
|---|---|---|
Redpanda upgrades and hotfixes |
✓ |
|
Cost management and attribution |
✓ |
✓ |
Software vulnerability remediation |
✓ |
|
Infrastructure vulnerability remediation |
✓ |
✓ |
IAM (roles, service accounts, access segmentation) |
✓ |
|
Compute |
✓ |
|
VPC (subnets, routing, firewall) |
✓ |
|
VPC peering |
✓ |
|
VPC private links (service endpoint) |
✓ |
|
VPC private links (consumer endpoint) |
✓ |
|
Local storage |
✓ |
|
Tiered Storage |
✓ |
|
Control plane |
✓ |
|
Access controls and audit |
✓ |
✓ |
Managed disaster recovery |
✓ |
|
Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks) |
✓ |
✓ |
Availability SLA |
✓ (subject to required access to customer resources) |
✓ |
Proactive threat detection |
✓ |
✓ |
Static secret rotation |
✓ |
✓ |
Incident response |
✓ |
|
Resilience verification |
✓ |
Redpanda Cloud architecture
When you first sign up for a Redpanda account, Redpanda creates an organization for you. Your organization contains all your Redpanda resources, including your clusters and networks. Within your organization, Redpanda creates a default namespace as a place to group your resources. You can rename this namespace, and you can create more namespaces. For example, you may want different namespaces for production and testing.
For high availability, Redpanda Cloud uses a control plane and data plane architecture.
-
Control plane: This is where most cluster management, operations, and maintenance takes place. The control plane enforces rules in the data plane.
-
Data plane: This is where your cluster lives. The term data plane is used interchangeably with cluster.
-
Agent: Redpanda uses an agent to manage the data plane from the control plane.
Clusters are configured and maintained in the control plane, but they remain available even if the network connection to the control plane is lost.
| In the user interface, when you’re at the organization (org) level or the namespace level, but you haven’t yet selected a cluster, you’re in the control plane. This is where you can select, create, and delete clusters, namespaces, and networks. When you’re at the cluster level working with topics, consumer groups, and connectors, you’re in the data plane. |
BYOC architecture
The following diagram shows a BYOC architecture, where you deploy the data plane in your own VPC. All network connections into the data plane take place through either a public endpoint or a VPC peering network connection. Sensitive data and credentials never leave the data plane.
A BYOC cluster is initially set up from the control plane. This is a two-step process performed by rpk cloud byoc apply:
-
You bootstrap a virtual machine (VM) in your VPC.
This VM spins up the agent and the required infrastructure. Redpanda assigns the necessary IAM policies required to run the agent and configures workload identity. That is, it configures independent IAM roles for each workload, with only the permissions each workload requires.
-
The agent communicates with the control plane to pull the cluster specifications.
After the agent is up and running, it connects to the control plane and starts dequeuing and applying cluster specifications that provision, configure, and maintain clusters. The agent is in constant communication with the control plane, receiving and applying cluster specifications and exchanging cluster metadata. Agents are authenticated and authorized through opaque and ephemeral tokens, and they have dedicated job queues in the control plane.
Agents also manage VPC peering networks. One agent manages each network resource at a time.
Redpanda Cloud vs self-hosted feature compatibility
Redpanda Cloud does not support the following self-hosted functionality:
-
mTLS
-
OpenID Connect (OIDC) authentication
-
Kerberos authentication
-
Data transforms
-
Remote Read Replicas
-
Redpanda Console topic documentation
-
Setting
auto_create_topics_enabled=truefor BYOC and dedicated clusters -
Admin API
-
The following
rpkcommands (which use the Admin API):-
rpk acl user -
rpk cluster health -
rpk cluster config -
rpk cluster license -
rpk cluster maintenance -
rpk cluster partitions -
rpk cluster self-test -
rpk cluster storage -
rpk generate app -
rpk redpanda -
rpk topic describe-storage(all otherrpk topiccommands are supported on both Redpanda Cloud and self-hosted)
-