What’s New

Redpanda now provides the framework to build and deploy inline transformations (data transforms) on data written to Redpanda topics, delivering processed and validated data to consumers in the format they expect. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms use WebAssembly (Wasm) engines inside a broker, which eliminates the need to manage a separate stream processing environment or use third-party tools.

You can enable OIDC for Redpanda to delegate the authentication process to an external identity provider (IdP) such as Okta, Azure AD, or on-premise Active Directory Federation Service (AD FS). When you enable OIDC, Redpanda does not need to manage user credentials directly, but can instead rely on the trusted authentication capabilities of IdPs. Redpanda’s implementation of OIDC provides SASL/OAUTHBEARER support for the Kafka API, and enables standard OIDC authentication across the Schema Registry, HTTP Proxy, and Admin APIs.

Redpanda’s audit logging supports fine-grained reporting of calls to the Kafka, HTTP Proxy, Schema Registry, and Admin APIs. Events include:

From 23.3.17, Redpanda has two new properties that provide finer control over cache management. These settings allow you to define specific thresholds for triggering cache trimming based on cache size and the number of objects, helping to optimize performance and prevent slow reads.

Auditing events can be stored in a topic, ensuring their retention for a specified period. They are protected against removal by all users, including Redpanda administrators. Auditing is compatible with the Open Cybersecurity Schema Framework (OCSF) and works with industry-standard tools, such as Splunk, Sumo Logic, and AWS.

With Tiered Storage enabled, you can do a whole cluster restore to recover all cluster data and metadata from a failed cluster to a new cluster. This is a simpler and cheaper alternative to active-active replication, for example with MirrorMaker 2. Use this recovery method to restore your applications to the latest functional state as quickly as possible.

Recovery mode lets you repair and revive a failed cluster that cannot start normally due to issues such as a system crash or out-of-memory errors. In recovery mode, Redpanda limits functionality to cluster configuration changes and other manual administrative actions so that you can repair the cluster.

Redpanda can now leverage data that has been uploaded to object storage to provide much faster commissioning and decommissioning of brokers. Instead of transferring all data from local storage to the reassigned replicas, the replicas can instead be initialized to rely more heavily on Tiered Storage for read requests. Aside from faster scaling up and down, this approach also reduces cross-AZ data transfer costs by limiting the amount of data replication between brokers during these operations.

Compaction follows a new algorithm, making it more efficient and similar to Kafka’s compaction. Compaction boundaries are no longer created between segments or Raft terms, and compaction is performed on data already uploaded to Tiered Storage as long as that data still exists in local storage. This improves the efficiency of compaction in both storage tiers.

Redpanda builds a map of keys to track the latest offset per key using multiple segments. The compaction process uses this key map to deduplicate keys within the compacted segments. It no longer stores full-fidelity keys, but instead maps SHA-256 hashes to offsets. The amount of memory that each shard may use when creating the key map for a partition is configurable. The default is 128 MiB.

As an alternative to Redpanda partition balancing, you now can explicitely change partition assignments with rpk cluster partitions move.

Each time a producer sends a message to a topic, Redpanda records the broker’s system date and time in the broker_timestamp property of the message. This feature helps maintain accurate retention policies, even when the message’s creation timestamp deviates from the broker’s time.

Redpanda now exposes metrics to help administrators manage their installed certificates. When queried, these metrics list details for all resources that have an installed certificate. This may include APIs, storage, or other assets. These metrics also support labels so that you can more readily report statistics on single resources. Configuring alerts on these metrics is a critical tool for managing certificate expiration and avoiding surprise outages.

The following rpk commands are new in this release:

The following commands have new flags:

The following commands have improved documentation or examples:

Upgrade Redpanda