Redpanda in Kubernetes

Kubernetes is a container orchestration tool that helps you manage Redpanda deployments using declarative configuration files called manifests. Kubernetes provides a standardized way of achieving the following:

  • High availability

  • Disaster recovery

  • Scalability

Redpanda Helm chart vs Redpanda Operator

To deploy Redpanda in Kubernetes, you can choose to use Helm for its simplicity, or combine Helm with the Redpanda Operator to leverage custom resource definitions (CRDs) for a more GitOps-friendly deployment process. Redpanda Operator is the recommended deployment option.

Feature Helm Helm + Redpanda Operator

Version management

Directly supported through Helm, enabling updates and rollbacks.

Managed through Helm.

Multi-tenant Kubernetes clusters

Supported. Helm allows you to deploy multiple independent Redpanda clusters by defining different Helm release names.

Supported. The Redpanda Operator enables a more declarative approach for managing multi-tenant clusters using the RedpandaList custom resource.

Dependency and configuration management

Managed directly through the Redpanda Helm chart.

Configurations defined in custom resources are passed to Helm, which then manages the dependencies and configurations.
If you are already using the Redpanda Helm chart and want to migrate to the latest Redpanda Operator to manage your deployment, see Migrate from the Redpanda Helm chart.

Helm

Helm is a package manager for Kubernetes, which simplifies the process of defining, installing, and upgrading Kubernetes applications. Helm uses charts, a collection of files that describe a related set of Kubernetes resources, to deploy applications in a Kubernetes cluster.

The Redpanda Helm chart generates and applies all the manifest files you need for deploying Redpanda in Kubernetes, including:

  • The StatefulSet to manage the Redpanda brokers.

  • A Headless ClusterIP Service for internal communication with the Redpanda cluster.

  • A NodePort Service for external communication with the Redpanda cluster.

The Redpanda Helm chart comes with default settings that enable a straightforward deployment out of the box. However, Helm also offers customization by allowing you to override these default values.

You can override the defaults in your own YAML files using the --values option or directly in the command line with the --set option. These changes in Helm chart values not only influence the configuration of the Redpanda cluster, but also determine what Kubernetes resources are deployed and how these resources are configured.

Helm and Redpanda Operator

The recommended option for deploying Redpanda in Kubernetes is a combination of Helm and the Redpanda Operator. While the Helm chart facilitates the deployment of Redpanda, the Redpanda Operator manages the lifecycle of Redpanda clusters. The operator uses Flux to automate the deployment of the Redpanda Helm chart through the HelmRelease resource.

Flux, when integrated with the Redpanda Operator, uses the HelmRelease resource to manage the lifecycle of the Redpanda Helm chart in the Kubernetes cluster. This integration streamlines the creation and management of Redpanda clusters, ensuring they are always in the desired state as defined by the operator’s custom resource definitions (CRDs).

CRDs for the Redpanda Operator include Flux CRDs. Flux manages the Redpanda Helm chart through the HelmRelease resource.

Here’s how it works with Flux:

  1. The Redpanda Operator defines the necessary CRDs for Redpanda. The Redpanda resource represents a Redpanda cluster that’s configured using values from the Redpanda Helm chart.

  2. When the Redpanda Operator detects changes to these Redpanda resources, it triggers Flux to reconcile the corresponding HelmRelease resource.

  3. Flux uses the HelmRelease resource to manage the lifecycle of the Redpanda Helm chart in the cluster.

If you already have Flux installed in your environment, you can disable the Flux HelmReleaseController and SourceController in the Redpanda Operator to prevent redundant reconciliations of HelmRelease resources. For details, see Deploy a Redpanda cluster.

Managed vs bare-metal

You can run Redpanda on managed Kubernetes services as well as in bare-metal environments. Managed Kubernetes services offer simpler deployment and maintenance, while bare-metal environments provide complete control and potential cost efficiencies.

Managed Kubernetes

Managed Kubernetes services manage one or more components of a Kubernetes cluster for you. Several cloud computing vendors provide this service, such as Google Cloud’s Google Kubernetes Engine (GKE) and Amazon Web Services' Elastic Kubernetes Service (EKS).

Managed Kubernetes platforms provide the following benefits:

  • Ease of deployment: Managed Kubernetes platforms allow you to provision cloud instances to serve as worker nodes. These instances are pre-configured with Kubernetes agent software and automatically join your Kubernetes cluster, making the process of deploying Redpanda simpler and more efficient.

  • Control plane maintenance: The managed service provider maintains and updates the control plane software, ensuring that it remains secure, reliable, and up-to-date.

  • Health monitoring and repairs: The health of the master nodes is continuously monitored, and repairs are made as necessary. This provides an additional level of confidence in the reliability of the platform.

However, you are still responsible for deploying and maintaining your Redpanda instances on the worker nodes.

Bare-metal Kubernetes environments

Bare-metal Kubernetes environments refer to any deployments where you are responsible for both the control plane and the worker nodes. Running Redpanda on bare-metal environments offers several advantages:

  • Complete control: With bare-metal Kubernetes, you have full control over every aspect of the deployment. Bare-metal deployments may be beneficial when your needs aren’t addressed by managed Kubernetes services.

  • Custom configuration: Bare-metal allows for granular control over the Kubernetes configuration, meaning you can fine-tune the environment.

  • Cost efficiency: Owning and operating your own hardware may prove to be more cost-effective.

Documentation conventions

The Kubernetes documentation follows these conventions:

  • Resource names: Kubernetes resources names, such as Service or PersistentVolume, are distinguished by the use of Pascal case. These are the names of resources when specified as a kind in manifest files.

  • Helm values: Helm values, such as storage.persistentVolume.enabled, are rendered in monospace font and written according to the JSON path specification.

Next steps

Whether you’re deploying locally or in the cloud, choose one of the following guides to get you started:

Or, you can explore our production workflow to learn more about the requirements and best practices.

Suggested reading

Suggested labs

Search all labs