What’s New

In release 24.1.5, Redpanda announced Redpanda Connect. This includes over 220 open source connectors, with a large collection of sources, sinks, and processors. The interface provides a lightweight and easy way to set up, debug, and manage connectors.

Write caching is a relaxed mode of acks=all that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. For clusters in development mode, write caching is enabled by default. For clusters in production mode, it is disabled by default.

Role-based access control (RBAC) provides an extension to Redpanda’s existing access control lists (ACLs). Using rpk or Redpanda Console, you can create roles and assign those roles as the authorized or denied party for an ACL. Every user you assign to that role inherits those privileges. RBAC provides a more flexible and efficient way to manage user privileges, especially with complex organizational hierarchies or large numbers of users.

Data transforms is now generally available for all Redpanda Community and Redpanda Enterprise users. This provides the framework to build and deploy inline transformations on data written to Redpanda topics using WebAssembly, delivering processed and validated data to consumers in the format they expect.

The following features have been added since the 23.3 beta release:

From 24.1.10, Redpanda has two new properties that provide finer control over cache management. These settings allow you to define specific thresholds for triggering cache trimming based on cache size and the number of objects, helping to optimize performance and prevent slow reads.

Redpanda now prioritizes balancing a topic’s partition replica count evenly across all brokers while it’s balancing the cluster’s overall partition count. Because different topics in a cluster can have vastly different load profiles, this better distributes the workload of the most heavily-used topics evenly across brokers. This feature is enabled by default with the partition_autobalancing_topic_aware property. After upgrading to 24.1, new topics are distributed in a topic-aware fashion, and the next partition rebalance (triggered by Continuous Data Balancing, by adding new brokers, or by calling the /v1/partitions/rebalance Admin API endpoint) evens out the partition counts on each broker.

Redpanda Tiered Storage now supports managed identities for Azure resources, similar to the existing support for IAM roles for self-hosted deployments on AWS and GCP. This provides a more secure alternative to static credentials, like access keys. Managed identities allow Redpanda clusters deployed on Azure virtual machines and on Azure Kubernetes Service (AKS) to access Azure Blob Storage (ABS) and other services without storing credentials in the cluster configuration.

A new property, fetch_read_strategy, by default sets new and migrated clusters with a non-polling strategy to fulfill fetch requests. The backend is signaled when a partition has new data, so it doesn’t need to repeatedly read from every partition in the fetch. This should improve fetch latency and CPU utilization for most workloads.

To revert to the previous polling strategy, run:

Before topic recovery or cluster restore, the properties cloud_storage_recovery_topic_validation_mode and cloud_storage_recovery_topic_validation_depth now check that partition manifests exist and confirm that there are no connection issues to object storage.

The following rpk commands are new in this release: