Redpanda Cloud Overview

Redpanda supports Dedicated Cloud, with clusters running in Redpanda Cloud, and also Bring Your Own Cloud (BYOC), with clusters running in your private cloud. Redpanda Cloud continuously monitors and maintains your Redpanda clusters along with the underlying Redpanda Cloud infrastructure to meet strict performance, availability, reliability, and security requirements. These clusters are also deployed with an integrated and dedicated Redpanda Console and, optionally, with a fully managed Kafka Connect and connectors cluster.

Cloud architecture

Redpanda Cloud uses a control plane and data plane architecture. The control plane configures and maintains Redpanda Cloud cluster agents, which are self-contained and independent of the control plane.

Control plane

The control plane is a separate entity that enforces rules in the data plane. The control plane is where a good deal of your Redpanda cluster management, operations, and maintenance takes place. While Redpanda Cloud clusters are configured and maintained in the control plane, they are self-contained and remain available even if the network connection to the control plane is lost.

Data plane

The data plane is where your Redpanda cluster resides in Redpanda Cloud; it is self-contained and managed by an agent that receives cluster specifications from the control plane. Redpanda Cloud clusters are interchangeably referred to as data planes. All network connections into the data plane take place through either a public endpoint or a Redpanda Cloud virtual private cloud (VPC) peering network connection.

Agent

The Redpanda Cloud agent, also known as the data plane agent, provisions, configures, and maintains all the resources in the Redpanda Cloud cluster, from the network all the way up to the Redpanda cluster and connectors infrastructure. Data plane resources include cloud infrastructure, Kubernetes, and data plane software artifacts.

Agents can be bootstrapped in two different ways: automatically by a machine or manually by a human. The human-driven agent bootstrap is designed for BYOC deployments. After the agent is up and running, it connects to the control plane and starts dequeuing and applying cluster specifications that provision, configure, and maintain Redpanda Cloud clusters.

The agent is in constant communication with the Redpanda Cloud control plane, receiving and applying cluster specifications and exchanging cluster metadata. Agents are authenticated and authorized through opaque and ephemeral tokens, and have dedicated job queues in the control plane. In BYOC, you can provision, configure, and maintain agents using rpk.

Redpanda Cloud agents also manage VPC peering networks. However, only one agent manages each network resource at a time.

The following diagram identifies the resources, operations, and connections that exist in and between the control plane and data plane.

Redpanda Cloud Security Architecture

Ways to provision clusters

Redpanda offers two ways to provision clusters in the cloud:

  • Dedicated Cloud: AWS or GCP

  • Bring Your Own Cloud (BYOC): AWS or GCP

Both methods offer the following features:

  • Single-zone or multi-zone availability

    • A single-zone cluster is deployed in one availability zone. If that availability zone fails, the cluster becomes unavailable.

    • A multi-zone cluster is deployed across three availability zones. This provides higher resiliency in the event of a failure in one of the zones.

  • Private networking using VPC peering

  • Ability to export metrics to a 3rd-party monitoring system

  • Managed connectors

  • Unlimited retention with Tiered Storage

Dedicated Cloud

With Dedicated Cloud, you host your data in Redpanda’s VPC, and Redpanda handles provisioning, operations, and maintenance. Dedicated clusters are single-tenant deployments launched from the Redpanda control plane. The Redpanda cluster is created in Redpanda’s VPC, and the endpoints are exposed so you can produce or consume events.

BYOC

With BYOC, you deploy Redpanda in your own VPC, so all data is contained in your own environment. There are no inbound connections from the control plane to the data plane. This provides an additional layer of security and isolation. Redpanda handles provisioning, monitoring, and upgrades. Setting up a BYOC cluster is initiated from the Redpanda control plane. This is a two-step process performed using rpk:

  1. You bootstrap a virtual machine (VM) in your VPC. This VM spins up the agent and the required infrastructure.

  2. The agent communicates with the control plane to pull the cluster specifications.

When provisioning BYOC, at the time an agent is deployed, Redpanda assigns the necessary IAM policies required to run the agent.

With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the most security, you can manage these resources yourself with a customer-managed VPC.

Cluster tiers

When you create a cluster, you select your throughput tier. For example, the following table lists the current AWS tier limits. Legacy tiers may have different limits. For more information, or for the current GCP limits, contact support.

Maximum ingress Maximum egress Maximum partitions Maximum connections

Tier 1

20 MBps

60 MBps

1,000

1,500

Tier 2

50 MBps

150 MBps

2,800

3,750

Tier 3

100 MBps

200 MBps

5,600

7,500

Tier 4

200 MBps

400 MBps

11,200

15,000

Tier 5

400 MBps

800 MBps

22,800

30,000

Redpanda Cloud vs self-hosted feature compatibility

Redpanda Cloud does not support the following self-hosted functionality:

  • mTLS

  • Remote Read Replicas

  • Kerberos authentication

  • Control plane SSO

  • Redpanda Console topic documentation

  • Admin API

  • The following rpk commands (which use the Admin API):

    • rpk acl user

    • rpk cluster health

    • rpk cluster config

    • rpk cluster license

    • rpk cluster maintenance

    • rpk cluster partitions

    • rpk cluster self-test

    • rpk cluster storage

    • rpk generate app

    • rpk redpanda

    • rpk topic describe-storage (all other rpk topic commands are supported on both Redpanda Cloud and self-hosted)

Next steps

Suggested reading

Suggested videos