24.1 23.3 23.2

What’s New

This topic includes new content added in version 24.1. For a complete list of all product updates, see the Redpanda release notes. See also:

Redpanda Connect

In release 24.1.5, Redpanda announced Redpanda Connect. This includes over 220 open source connectors, with a large collection of sources, sinks, and processors. The interface provides a lightweight and easy way to set up, debug, and manage connectors.

Write caching

Write caching is a relaxed mode of acks=all that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. For clusters in development mode, write caching is enabled by default. For clusters in production mode, it is disabled by default.

RBAC

Role-based access control (RBAC) provides an extension to Redpanda’s existing access control lists (ACLs). Using rpk or Redpanda Console, you can create roles and assign those roles as the authorized or denied party for an ACL. Every user you assign to that role inherits those privileges. RBAC provides a more flexible and efficient way to manage user privileges, especially with complex organizational hierarchies or large numbers of users.

Data transforms

Data transforms is now generally available for all Redpanda Community and Redpanda Enterprise users. This provides the framework to build and deploy inline transformations on data written to Redpanda topics using WebAssembly, delivering processed and validated data to consumers in the format they expect.

The following features have been added since the 23.3 beta release:

  • Data transforms can now output to multiple topics. For example, you can distribute different message types to multiple event streams, or you can write valid records to an output topic and invalid records to a dead-letter queue.

  • There is a new Rust SDK for data transforms. This complements the existing Golang SDK.

  • Data transform logs are now stored in an internally-managed topic and are accessible via rpk and Redpanda Console.

Topic-aware partition balancing

Redpanda now prioritizes balancing a topic’s partition replica count evenly across all brokers while it’s balancing the cluster’s overall partition count. Because different topics in a cluster can have vastly different load profiles, this better distributes the workload of the most heavily-used topics evenly across brokers. This feature is enabled by default with the partition_autobalancing_topic_aware property. After upgrading to 24.1, new topics are distributed in a topic-aware fashion, and the next partition rebalance (triggered by Continuous Data Balancing, by adding new brokers, or by calling the /v1/partitions/rebalance Admin API endpoint) evens out the partition counts on each broker.

Managed identities for Azure

Redpanda Tiered Storage now supports managed identities for Azure resources, similar to the existing support for IAM roles for self-hosted deployments on AWS and GCP. This provides a more secure alternative to static credentials, like access keys. Managed identities allow Redpanda clusters deployed on Azure virtual machines and on Azure Kubernetes Service (AKS) to access Azure Blob Storage (ABS) and other services without storing credentials in the cluster configuration.

Improved fetch latency

A new property, fetch_read_strategy, by default sets new and migrated clusters with a non-polling strategy to fulfill fetch requests. The backend is signaled when a partition has new data, so it doesn’t need to repeatedly read from every partition in the fetch. This should improve fetch latency and CPU utilization for most workloads.

To revert to the previous polling strategy, run:

rpk cluster config set fetch_read_strategy polling

Validation before topic recovery or cluster restore

Before topic recovery or cluster restore, the properties cloud_storage_recovery_topic_validation_mode and cloud_storage_recovery_topic_validation_depth now check that partition manifests exist and confirm that there are no connection issues to object storage.

New commands

The following rpk commands are new in this release:

Documentation enhancements

  • Redpanda Labs provides examples, guidance, best practices, and ideas for how to use Redpanda in your own project.

  • Redpanda Connect documentation is added as a new dropdown option, similar to Redpanda Labs.

  • Improved search lets you filter for blogs, videos, labs, and documentation for all supported versions.

  • New side-by-side codeblocks let you easily compare code examples.

  • Improved listing of changes to rpk commands in deprecated features.

  • List of Redpanda integrations built and supported by our partners.

Next steps