Migrate from Strimzi to Redpanda Operator

In this guide, you’ll explore the key differences between Strimzi and Redpanda Operator, and you’ll find examples for migrating Strimzi custom resources to Redpanda.

The information in this guide is based on version 0.40.0 of the Strimzi Operator.

Why Redpanda?

Redpanda is a Kafka-compatible event streaming platform that aims to simplify data streaming by eliminating external dependencies like ZooKeeper and offering a leaner architecture. See When to choose Redpanda instead of Apache Kafka.

About Strimzi and Redpanda Operator in Kubernetes

Both Strimzi and the Redpanda Operator harness the power of Kubernetes to provide tools for creating and managing Kafka clusters in Kubernetes environments:

  • Strimzi is an open-source project that simplifies the deployment and management of Apache Kafka clusters on Kubernetes.

  • The Redpanda Operator simplifies the deployment and management of Redpanda clusters on Kubernetes.

The Redpanda Operator uses Flux to automate the deployment of the Redpanda Helm chart through the HelmRelease resource. For more details, see Redpanda in Kubernetes.

CRDs for the Redpanda Operator include Flux CRDs. Flux manages the Redpanda Helm chart through the HelmRelease resource.

Notable differences

These are some notable differences between features and functionality in Strimzi and Redpanda in Kubernetes.

Security

The Redpanda Helm chart emphasizes security out of the box, with the following defaults:

  • TLS enabled for the Kafka API, HTTP Proxy, and inter-broker communication (RPC).

  • Certificate management through cert-manager by default.

Both Strimzi and the Redpanda Operator currently do not support Kerberos for Kafka.

See Security for Redpanda in Kubernetes.

Storage

Redpanda performs best with direct-attached storage (NVMe) to ensure high performance. To minimize latency, use of Kubernetes local PersistentVolumes.

See Kubernetes Cluster Requirements and Recommendations.

JVM

Redpanda does not require a JVM because it is written in C++.

Zookeeper

Redpanda does not require ZooKeeper. Instead, Redpanda brokers run the Raft consensus algorithm for cluster coordination.

Cruise control

Redpanda manages partition distribution and load balancing, eliminating the need for external tools like cruise control. See Cluster Balancing.

Networking

The Redpanda Helm chart has built-in support for the following networking options:

  • NodePort for direct access to services from outside the cluster using ports on each node.

  • Load Balancer for distributing external traffic to the cluster nodes, typically managed by cloud providers.

Ingress and route configurations are not available in the Redpanda Helm chart. To use these Services, you can disable the default NodePort Service and deploy your own custom Service.

See Networking and Connectivity in Kubernetes.

Observability

Redpanda exposes metrics to an HTTP endpoint by default, facilitating straightforward integration with monitoring tools such as Prometheus.

Strimzi includes built-in support for distributed tracing with OpenTelemetry. However, the Redpanda Helm chart lacks support for distributed tracing.

See Monitor Redpanda in Kubernetes and Public Metrics.

Custom resources

The deployment of Kafka components onto a Kubernetes cluster using both Strimzi and Redpanda Operator is configurable through custom resources. These resources are instances of APIs introduced by custom resource definitions (CRDs), which extend Kubernetes resources.

CRDs act as configuration instructions to describe the custom resources in a Kubernetes cluster. CRDs also allow resources to benefit from native Kubernetes features like CLI accessibility and configuration validation.

Both Strimzi and the Redpanda Operator offer CRDs for creating and managing components as Kubernetes resources. Some custom resources in Strimzi do not have an equivalent custom resource in Redpanda.

Strimzi custom resource Redpanda equivalent

Kafka

Redpanda custom resource

KafkaBridge

Built-in HTTP proxy

KafkaConnect

Available as a Helm chart

KafkaMirrorMaker2

Included in the Connectors Helm chart

KafkaNodePool

No support for heterogeneous brokers

KafkaTopic

Topic custom resource

KafkaUser

Users are managed through cluster configuration

Migrate to Redpanda Operator

To ensure a smooth migration:

Plan data and topic migration

When migrating from Strimzi to Redpanda Operator, plan your data and topic migration to ensure data integrity and minimal downtime:

  • Back up data: Before starting the migration, ensure that all data in Kafka topics is backed up.

  • Migrate topics: Review the configuration of your existing Kafka topics, including partition count, replication factor, and any custom configurations like retention policies. This information is essential for migrating these topics to Redpanda. See Migrate the KafkaTopic resource.

  • Transfer data: Determine the most suitable method for data transfer. For example, you can use MirrorMaker2 to replicate data from Kafka to Redpanda in real-time. This method is useful if you need to keep the source system online during the migration. See Migrate the KafkaMirrorMaker2 resource.

  • Connect clients: Connect client applications to Redpanda.

Migrate Strimzi resources to Redpanda

Migrating from Strimzi to Redpanda involves converting Strimzi custom resources into their corresponding forms for the Redpanda Operator. This process ensures that your Kafka configurations and setups are correctly translated and optimized for Redpanda.

These example Strimzi manifests are for version 0.40.0 of the Strimzi Operator.

Migrate the Kafka resource

This section provides an example of how to translate configuration from a Strimzi Kafka resource to a Redpanda resource.

Strimzi
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.7.0 (1)
    replicas: 1 (2)
    listeners: (3)
      - name: plain
        port: 9093
        type: internal
        tls: false
      - name: tls
        port: 9094
        type: internal
        tls: true
    config: (4)
      default.replication.factor: 1
    storage: (5)
      type: ephemeral
    rack: (6)
      topologyKey: topology.kubernetes.io/zone
    config:
      replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector
  zookeeper: (7)
    replicas: 3
    storage:
      type: ephemeral
  entityOperator: (8)
    topicOperator: {}
    userOperator: {}
Redpanda
apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    image:
      tag: v23.3.11 (1)
    statefulset:
      replicas: 1 (2)
    listeners: (3)
      kafka:
        port: 9093
        authenticationMethod:
        tls:
          enabled: false
    config: (4)
      cluster:
        default_topic_replications: 1
    storage: (5)
      hostPath: ""
      persistentVolume:
        enabled: false
    rackAwareness: (6)
      enabled: true
      nodeAnnotation: 'topology.kubernetes.io/zone'
    serviceAccount:
      create: true
    rbac:
      enabled: true
1 Versioning: Strimzi refers to Kafka versions, while Redpanda uses its own versioning scheme.
2 Replicas: Configures the number of cluster replicas to 1 in both resources, directly translating the desired number of broker nodes from Strimzi to Redpanda.
3 Listeners: Strimzi allows defining multiple internal Kafka listeners, whereas Redpanda allows for only one internal listener with a singular port definition.
4 Configuration: Strimzi’s default.replication.factor setting translates to default_topic_replications in Redpanda, aligning Kafka cluster configurations between the two.
5 Storage: Both platforms support ephemeral storage for non-persistent environments, typically used for testing or development. See Storage for Redpanda in Kubernetes.
6 Rack awareness: Both platforms support high availability and fault tolerance by spreading replicas across different physical locations. Strimzi uses topologyKey, and Redpanda uses rackAwareness with nodeAnnotation. See Enable Rack Awareness in Kubernetes.
7 ZooKeeper: Necessary for cluster management in Strimzi but not used in Redpanda, which employs the Raft consensus algorithm for managing cluster state.
8 Entity Operator: Manages Kafka topics and users through separate operators in Strimzi. Redpanda handles topic management through the Topic custom resource but does not support user management in CRDs.

Migrate the KafkaTopic resource

In Strimzi, a single KafkaTopic resource is used to manage a single topic in a single Kafka cluster. In the following example, the resource has a label strimzi.io/cluster with the name of the target Kafka cluster. The Strimzi Operator communicates with this cluster and ensures that the specified topic is created or updated according to the desired configuration.

In Redpanda, the Topic resource is also used to manage a single topic in a single Redpanda cluster. Like the Strimzi Topic Operator, the Redpanda Topic Controller is unidirectional. The controller reconciles topic changes in only one direction: from Kubernetes to Redpanda. For more details, see: Manage Topics with the Redpanda Operator.

Previous versions of the Strimzi Topic Operator supported bidirectional topic management. Redpanda Operator does not support bidirectional topic management.
Strimzi
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic (1)
  labels:
    strimzi.io/cluster: my-kafka-cluster (2)
spec:
  partitions: 3 (3)
  replicas: 3 (4)
Redpanda
apiVersion: cluster.redpanda.com/v1alpha1
kind: Topic
metadata:
  name: my-topic (1)
spec:
  kafkaApiSpec: (2)
    brokers:
      - "redpanda-0.redpanda.<namespace>.svc.cluster.local:9093"
      - "redpanda-1.redpanda.<namespace>.svc.cluster.local:9093"
      - "redpanda-2.redpanda.<namespace>.svc.cluster.local:9093"
    tls:
      caCertSecretRef:
        name: "redpanda-default-cert"
        key: "ca.crt"
  partitions: 3 (3)
  replicationFactor: 3 (4)
1 Topic name: Both configurations identify the Kafka topic by the same name, ensuring consistency across migration.
2 Cluster reference: Strimzi uses labels to link the topic to the specific Kafka cluster, whereas Redpanda uses a kafkaApiSpec block, explicitly defining the brokers and security settings.
3 Partitions: Both platforms maintain the same number of partitions for the topic, facilitating a direct translation of partition configuration.
4 Replicas: The number of replicas is set to 3 in both cases, ensuring high availability and data redundancy during and after migration.

Migrate the KafkaUser resource

The Redpanda Operator does not support a custom resource for Kafka users. For details on user authentication in Redpanda, see Configure Authentication for Redpanda in Kubernetes.

Migrate the KafkaConnect resource

The Redpanda Operator does not support a custom resource to define Kafka Connect deployments. Redpanda provides support for Kafka Connect through a separate Connectors Helm chart.

See Deploy Redpanda Connectors in Kubernetes.

Migrate the KafkaBridge resource

Redpanda includes a built-in HTTP proxy on each broker, enabling direct HTTP-based interactions without the need for a separate bridge component. For details, see Use Redpanda with the HTTP Proxy API. For the API reference, see HTTP Proxy API.

Migrate the KafkaMirrorMaker2 resource

The Redpanda Operator does not support a custom resource for MirrorMaker2. Redpanda offers a separate Helm chart that includes Kafka Connect and MirrorMaker2. For information on how to use MirrorMaker2 with Redpanda in Kubernetes:

  1. Deploy Redpanda Connectors in Kubernetes.

  2. Create and Manage Connectors in Kubernetes.

Migrate the KafkaNodePool resource

The Redpanda Operator does not have an equivalent for KafkaNodePool resources in Strimzi. The Redpanda Helm chart allows you to deploy only homogenous broker configuration in a single Redpanda cluster, unlike the heterogeneous configurations available in KafkaNodePools.

Brokers in Redpanda are uniformly configured according to the specifications in RedpandaClusterSpec. Given this difference, you must adapt your Kafka setup to a unified broker configuration model by standardizing the broker settings that were previously varied across different KafkaNodePool resources.

Adjust monitoring and alerting

Adjust your monitoring and alerting setup to ensure visibility into the Redpanda environment and maintain operational stability.

  • Configure monitoring tools: If you use tools like Prometheus for monitoring, reconfigure them to scrape metrics from Redpanda. Redpanda exposes metrics at an HTTP endpoint, which might require changes to your Prometheus scraping configurations. See Monitor Redpanda in Kubernetes and Public Metrics.

  • Update dashboards: Update or recreate Grafana dashboards to reflect the metrics provided by Redpanda. This might involve adjusting metric names and labels to align with those emitted by Redpanda. See Generate Grafana dashboard.

  • Set up new alerts: Review and revise alerting rules to ensure they are relevant for the Redpanda environment. This includes setting thresholds that are appropriate for the performance and behavior of Redpanda as compared to Kafka.

  • Monitor logs: Integrate Redpanda with your log management solutions. Ensure that logs emitted by Redpanda are collected, stored, and indexed effectively, allowing for easy querying and monitoring.

Engage with the Redpanda community

Leverage the Redpanda community Slack for support during and after your migration. The community can offer insights, best practices, and assistance in optimizing your streaming platform.

Next steps

See the following resources: