What’s New

This topic summarizes new content added in version 23.3. For a complete list of all product updates, see the Redpanda release notes.

See also: What’s New in Redpanda Cloud

Data transforms

Redpanda now provides the framework to build and deploy inline transformations (data transforms) on data written to Redpanda topics, delivering processed and validated data to consumers in the format they expect. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms use WebAssembly (Wasm) engines inside a broker, which eliminates the need to manage a separate stream processing environment or use third-party tools.

OIDC authentication

You can enable OIDC for Redpanda to delegate the authentication process to an external identity provider (IdP) such as Okta, Azure AD, or on-premise Active Directory Federation Service (AD FS). When you enable OIDC, Redpanda does not need to manage user credentials directly, but can instead rely on the trusted authentication capabilities of IdPs. Redpanda’s implementation of OIDC provides SASL/OAUTHBEARER support for the Kafka API, and enables standard OIDC authentication across the Schema Registry, HTTP Proxy, and Admin APIs.

Audit logging

Redpanda’s audit logging supports fine-grained reporting of calls to the Kafka, HTTP Proxy, Schema Registry, and Admin APIs. Events include:

  • Changes to clusters, users, topics, ACLs, or consumer groups

  • Any time a user is authenticated

  • Any time a topic is written to or read from (requires explicit opt-in)

  • HTTP requests for the Schema Registry, HTTP Proxy, and Admin APIs

Enhanced cache trimming

From 23.3.17, Redpanda has two new properties that provide finer control over cache management. These settings allow you to define specific thresholds for triggering cache trimming based on cache size and the number of objects, helping to optimize performance and prevent slow reads.

Topic-aware partition balancing

Auditing events can be stored in a topic, ensuring their retention for a specified period. They are protected against removal by all users, including Redpanda administrators. Auditing is compatible with the Open Cybersecurity Schema Framework (OCSF) and works with industry-standard tools, such as Splunk, Sumo Logic, and AWS.

Whole cluster restore

With Tiered Storage enabled, you can do a whole cluster restore to recover all cluster data and metadata from a failed cluster to a new cluster. This is a simpler and cheaper alternative to active-active replication, for example with MirrorMaker 2. Use this recovery method to restore your applications to the latest functional state as quickly as possible.

Recovery mode

Recovery mode lets you repair and revive a failed cluster that cannot start normally due to issues such as a system crash or out-of-memory errors. In recovery mode, Redpanda limits functionality to cluster configuration changes and other manual administrative actions so that you can repair the cluster.

Fast commission and decommission

Redpanda can now leverage data that has been uploaded to object storage to provide much faster commissioning and decommissioning of brokers. Instead of transferring all data from local storage to the reassigned replicas, the replicas can instead be initialized to rely more heavily on Tiered Storage for read requests. Aside from faster scaling up and down, this approach also reduces cross-AZ data transfer costs by limiting the amount of data replication between brokers during these operations.

Compaction improvements

Compaction follows a new algorithm, making it more efficient and similar to Kafka’s compaction. Compaction boundaries are no longer created between segments or Raft terms, and compaction is performed on data already uploaded to Tiered Storage as long as that data still exists in local storage. This improves the efficiency of compaction in both storage tiers.

Redpanda builds a map of keys to track the latest offset per key using multiple segments. The compaction process uses this key map to deduplicate keys within the compacted segments. It no longer stores full-fidelity keys, but instead maps SHA-256 hashes to offsets. The amount of memory that each shard may use when creating the key map for a partition is configurable. The default is 128 MiB.

Move partitions

As an alternative to Redpanda partition balancing, you now can explicitely change partition assignments with rpk cluster partitions move.

Broker time retention

Each time a producer sends a message to a topic, Redpanda records the broker’s system date and time in the broker_timestamp property of the message. This feature helps maintain accurate retention policies, even when the message’s creation timestamp deviates from the broker’s time.

Monitor TLS certificates

Redpanda now exposes metrics to help administrators manage their installed certificates. When queried, these metrics list details for all resources that have an installed certificate. This may include APIs, storage, or other assets. These metrics also support labels so that you can more readily report statistics on single resources. Configuring alerts on these metrics is a critical tool for managing certificate expiration and avoiding surprise outages.

Redpanda Console updates

  • You can perform Schema Registry operations in Redpanda Console, such as registering a schema, creating a new version of it, and configuring compatibility. The Schema Registry page lists verified schemas, including their serialization format and versions.

  • Redpanda Console automatically tries to deserialize incoming messages and show them in human-readable format. It tests different deserialization strategies until it finds one with no errors. If no deserialization attempts are successful, Redpanda Console renders the byte array in a hex viewer and displays troubleshooting information.

rpk updates

The following rpk commands are new in this release:

  • rpk registry: This includes commands to interact with the Schema Registry.

  • rpk transform: This includes commands to develop, deploy, and manage data transforms.

  • rpk cluster partitions This includes commands to move partitions, verify that the reassignment completes, and cancel moves.

  • rpk cluster partitions unsafe recover: This command allows you to unsafely recover all data adversely affected by the loss of brokers.

  • rpk cluster txn This includes commands to observe transactions and producers.

The following commands have new flags:

The following commands have improved documentation or examples:

Deprecated features

  • The delete_retention_ms property has been deprecated. Use log_retention_ms instead.

  • The rpk cluster storage recovery command was renamed to rpk cluster storage restore. You can still use rpk cluster storage recovery as an alias.

  • As of v23.3.2, the space_management_enable_override property, added as a workaround for prior versions of Redpanda, is no longer required. Redpanda now ignores this property.

Documentation enhancements

  • A new glossary provides definitions helpful for working with Redpanda. Terms with hover text and a link to the glossary entry render with a dotted line; for example, controller broker.

  • Documentation search now includes the option to search Redpanda blogs and videos.

  • Rolling restarts involve restarting one broker at a time while the remaining brokers in your cluster continue running. The documentation has been clarified for restarts and upgrades.

  • CPU profiler: There is a new Admin API endpoint to query the CPU profiler for troubleshooting.

Next steps

Upgrade Redpanda