Provision a Serverless Cluster

Beta

Redpanda Serverless is the fastest and easiest way to start event streaming in the cloud.

What is Redpanda Serverless?

Redpanda Serverless is a multi-tenant, globally-distributed managed streaming service (Kafka API) that abstracts users from scaling and operational concerns. It applies usage-based billing, so you only pay for what you consume. Serverless clusters provide data isolation and data residency:

  • Your data is hosted on Redpanda Cloud clusters shared among multiple tenants. Other tenants never access your data.

  • Your data is stored in the Redpanda Cloud account, in the region you select. Data never leaves that region.

Serverless vs Dedicated and BYOC

Redpanda Serverless is a good fit for the following:

  • Starter and growing workloads

  • Spiky workloads (development environments, systems that only occasionally get busy, or workloads that come and go)

  • Fast and dynamic cluster creation: you can use a Serverless cluster as an isolated container for topics

Because a Serverless cluster is created in under one second, you can surface it in your applications (for example, for tenant isolation).

If necessary, workloads can be migrated from Redpanda Serverless to Redpanda Dedicated or BYOC clusters. Consider Dedicated or BYOC for the following:

  • Control over cluster settings

  • Control over the deployment

  • Private networking

  • Workloads with consistently-high throughput

Deployment and networking

Redpanda Serverless currently is available in certain AWS regions. We expect your applications to be deployed in the same AWS region. For best performance, select the region closest to your applications. Serverless is not guaranteed to be pinned to a particular availability zone within that region.

Your applications talk to Serverless over the public internet using encrypted connections. Serverless does not support private networking (that is, VPC peering to talk over private IPs). Use Redpanda Dedicated clusters for private networking.

Limitations

These baseline limits are subject to change after the Serverless Beta release.

Each Serverless cluster has the following limits:

  • Ingress: up to 10 MBps, 0.5 MBps guaranteed

  • Egress: up to 30 MBps, 1.5 MBps guaranteed

  • Partitions: 100

  • Max message size: 20 MB

  • Max retention: unlimited

  • Storage: unlimited

  • Users: 30

  • ACLs: 120

  • Consumer groups: 200

Redpanda Serverless is opinionated about Kafka configurations. For example, automatic topic creation is disabled. Some systems expect the Kafka service to automatically create topics when a message is produced to a topic that doesn’t exist. You can create topics in the Redpanda Cloud Topics page or with rpk topic create.

Free trial

Start using Redpanda Serverless Beta by joining the free trial. To ask questions, give feedback, or extend your trial, use the #serverless Community Slack channel, or contact support.

Prerequisites

  1. Sign up for a Beta trial of Redpanda Serverless.

  2. Make sure you have the latest version of rpk. See Install rpk.

Credits and limits

Each Beta trial supports five Serverless clusters. Redpanda issues 100 free credits, which is enough to run Redpanda with reasonable throughput. These credits expire in 14 days. Usage is based on the following:

  • Ingress: 0.1 credit per GB in (writes)

  • Egress: 0.1 credit per GB out (reads)

  • Partitions: 0.003 credit (metered in 10 min fractions) per partition/hour

  • Storage: 0.1 credit (metered in 10 min fractions) per GB stored/month

After either the 14 day trial expires or the 100 credits expire, the clusters move into a suspended state, and you won’t be able to access your data in either the Redpanda Cloud UI or with the Kafka API. There is a 14 day grace period following the end of the trial, in case you want to extend it. After that, the data is permanently deleted. To extend your trial, use the #serverless Community Slack channel, or contact support.

Get started

For the Beta release, you’ll receive an email to activate your Serverless trial.

  1. In your Welcome email, click to sign up.

    Redpanda prepares a Redpanda Cloud account for you with a "Hello, World!" Serverless cluster in AWS.

  2. To log in to Redpanda Cloud, run the following command in your terminal:

    rpk cloud login

  3. Follow the CLI output for next steps.

Interact with your cluster

Similar to BYOC and Dedicated clusters, you can interact with your Serverless cluster with rpk CLI commands or with the Redpanda Cloud UI. You can add team members and grant them access with ACLs on the Security page.

Here are some helpful rpk commands:

  • rpk cloud login: Use this to log in to Redpanda Cloud or to refresh the session.

  • rpk topic: Use this to manage topics, produce data, and consume data.

  • rpk profile print: Use this to view rpk configuration and see the URL for your Serverless cluster.

  • rpk acl user: Use this to manage users and permissions.

Run a demo application

Run rpk generate app to generate a sample application to connect with Redpanda.

The Redpanda Cloud Overview page lists your bootstrap server URL and security settings in the How to connect - Kafka API tab. In the same section, you can click Go, Python, or Node.js for code examples to create a topic and produce and consume messages.

Supported features

Serverless clusters work with all Kafka clients. For more information, see Kafka Compatibility.

Serverless clusters support all major Apache Kafka messages for managing topics, producing/consuming data (including transactions), managing groups, managing offsets, and managing ACLs. (User management is available in the Redpanda Cloud UI or with rpk acl.)

Unsupported features

Redpanda Serverless supports the Kafka API. The Redpanda Admin, HTTP Proxy, and Schema Registry APIs are not exposed. You can use an external schema registry.

The following features are not yet supported in Serverless clusters:

  • Managed connectors

  • Data transforms