Cluster Configuration Properties

Capacity (in number of txns) of an abort index segment.

Each partition tracks the aborted transaction offset ranges to help service client requests. If the number of transactions increases beyond this threshold, they are flushed to disk to ease memory pressure. Then they’re loaded on demand. This configuration controls the maximum number of aborted transactions before they are flushed to disk.

Interval, in milliseconds, at which Redpanda looks for inactive transactions and aborts them.

Whether Admin API clients must provide HTTP basic authentication headers.

Enable aggregation of metrics returned by the /metrics endpoint. Aggregation can simplify monitoring by providing summarized data instead of raw, per-instance metrics. Metric aggregation is performed by summing the values of samples by labels and is done when it makes sense by the shard and/or partition labels.

The amount of time since the last broker status heartbeat. After this time, a broker is considered offline and not alive.

Size of direct write operations to disk in bytes. A larger chunk size can improve performance for write-heavy workloads, but increase latency for these writes as more data is collected before each write operation. A smaller chunk size can decrease write latency, but potentially increase the number of disk I/O operations.

Defines the number of bytes allocated by the internal audit client for audit messages. When changing this, you must disable audit logging and then re-enable it for the change to take effect. Consider increasing this if your system generates a very large number of audit records in a short amount of time.

Enables or disables audit logging. When you set this to true, Redpanda checks for an existing topic named _redpanda.audit_log. If none is found, Redpanda automatically creates one for you.

List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin.

List of user principals to exclude from auditing.

List of topics to exclude from auditing.

Defines the policy for rejecting audit log messages when the audit log queue is full. If set to 'permit', then new audit messages are dropped and the operation is permitted. If set to 'reject', then the operation is rejected.

Defines the number of partitions used by a newly-created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics.

Defines the replication factor for a newly-created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. Setting this value is optional. If a value is not provided, Redpanda will use the value specified for internal_topic_replication_factor.

Interval, in milliseconds, at which Redpanda flushes the queued audit log messages to the audit log topic. Longer intervals may help prevent duplicate messages, especially in high throughput scenarios, but they also increase the risk of data loss during shutdowns where the queue is lost.

Defines the maximum amount of memory in bytes used by the audit buffer in each shard. Once this size is reached, requests to log additional audit messages will return a non-retryable error. Limiting the buffer size per shard helps prevent any single shard from consuming excessive memory due to audit log messages.

Use Redpanda’s internal communication system to write audit logs. When disabled, Redpanda uses a Kafka client to write audit logs instead.

Allow automatic topic creation. To prevent excess topics, this property is not supported on Redpanda Cloud BYOC and Dedicated clusters. You should explicitly manage topic creation for these Redpanda Cloud clusters.

If you produce to a topic that doesn’t exist, the topic will be created with defaults if this property is enabled.

Introduced in v26.1.1

When enabled, the reconciler tolerates missing L0 extent objects (404 errors) during materialization. Failed extents are skipped, producing L1 state with empty offset ranges where deleted data was. Use this to recover partitions after accidental deletion of live extent objects.

Introduced in v26.1.1

How often to trigger background compaction for cloud topics.

Introduced in v26.1.1

Maximum number of bytes that may be used on each shard by cloud topics compaction key-offset maps.

Introduced in v26.1.1

Maximum size in bytes for L1 objects produced by cloud topics compaction.

Introduced in v26.1.1

Disables the level-zero garbage collector in cloud topics. This property exists to simplify testing and shouldn’t be set in production.

Introduced in v26.1.1

Disables the metastore flush loop in cloud topics. The property exists to simplify testing of read replicas and shouldn’t be set in production.

Disables the cloud topics reconciliation loop. Disabling the loop can negatively impact performance and stability of the cluster.

Enable Cloud Topics for the cluster. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.

Introduced in v25.3.3

The interval, in milliseconds, at which the cluster epoch is incremented.

The cluster epoch is a frozen point in time of the committed offset of the controller log, used to coordinate partition creation and track changes in Tiered Storage. This property controls how frequently the epoch is refreshed. More frequent updates provide finer-grained coordination but may increase overhead.

Decrease this interval if you need more frequent epoch updates for faster coordination in Tiered Storage operations, or increase it to reduce coordination overhead in stable clusters.

Introduced in v25.3.3

The duration, in milliseconds, for which a cluster-wide epoch is cached locally on each broker.

Caching the epoch locally reduces the need for frequent coordination with the controller. This property controls how long each broker can use a cached epoch value before fetching the latest value.

Increase this value to reduce coordination overhead in clusters with stable workloads. Decrease it if you need brokers to react more quickly to epoch changes in Tiered Storage.

Introduced in v26.1.1

The duration of time that a node can use the exact same epoch.

Introduced in v26.1.1

Enables fetch debouncing in cloud topics. This mechanism guarantees that the broker fetches every object only once improving the performance and lowering the cost.

Introduced in v26.1.1

The interval at which the L0 garbage collector checks cluster health. GC will not proceed while the cluster is unhealthy.

Introduced in v26.1.1

The byte interval at which index entries are created within long term storage objects for cloud topics. Index entries are stored in the object metadata and enable efficient seeking by offset or timestamp within a partition. Lower values produce more index entries (better seek granularity) at the cost of a larger footer.

Introduced in v26.1.1

Delay before deleting stale long term files, allowing concurrent readers (e.g. read replica topics) to finish reading them before removal.

Introduced in v26.1.1

Time interval at which long term storage metadata is flushed to object storage.

Time interval after which data is garbage collected from long term storage.

Introduced in v26.1.1

Timeout for applying a replicated write batch to the local LSM database. This may take longer than usual when L0 compaction is behind and writes are being throttled.

Introduced in v26.1.1

Timeout for L1 metastore Raft replication and waiting for the STM to apply the replicated write batch.

Introduced in v26.1.1

Number of partitions for the cloud topics metastore topic, used to spread metastore load across the cluster. Higher values allow more parallel metadata operations but reduce the amount of work each partition can batch together. Only takes effect when the metastore topic is first created.

Introduced in v26.1.1

Enable parallel fetching in cloud topics. This mechanism improves the throughput by allowing the broker to download data needed by the fetch request using multiple shards.

Introduced in v26.1.1

Time-to-live for pre-registered L1 objects before they are expired.

The size limit for the object size in cloud topics. When the amount of data on a shard reaches this limit, an upload is triggered.

Threshold for the object cardinality in cloud topics. When the number of partitions in waiting for the upload reach this limit, an upload is triggered.

Introduced in v26.1.1

Maximum number of concurrent raft replication requests for producers without a producer ID (idempotency disabled). Limits how many no-PID writes can proceed past the producer queue into raft simultaneously.

Time interval after which the upload is triggered.

Introduced in v26.1.1

Maximum number of in-flight write requests per shard in the cloud topics write pipeline. Requests that exceed this limit are queued until a slot becomes available.

Configuration property: cloud_topics_reconciliation_interval

Introduced in v26.1.1

Maximum reconciliation interval for adaptive scheduling.

Introduced in v26.1.1

Maximum size in bytes for L1 objects produced by the reconciler. With the default target fill ratio of 0.8, this gives an effective target object size of 64 MiB.

Introduced in v26.1.1

Minimum reconciliation interval for adaptive scheduling. The reconciler will not run more frequently than this.

Introduced in v26.1.1

Maximum number, per shard, of concurrent objects built by reconciliation

Introduced in v26.1.1

Blend factor for slowing down reconciliation (0.0 to 1.0). Higher values mean reconciliation lowers its frequency faster when trying to find a frequency that produces well-sized objects. Generally this should be lower than the speedup blend, because reconciliation has less opportunities to adapt its frequency when it runs less frequently.

Introduced in v26.1.1

Blend factor for speeding up reconciliation (0.0 to 1.0). Higher values mean reconciliation increases its frequency faster when trying to find a frequency that produces well-sized objects.

Introduced in v26.1.1

Target fill ratio for L1 objects. The reconciler adapts its interval to produce objects at approximately this fill level (0.0 to 1.0).

Introduced in v25.3.3

The interval, in milliseconds, between invocations of the L0 garbage collection work loop when no progress is being made or errors are occurring.

L0 (level-zero) objects are short-term data objects in Tiered Storage that are periodically garbage collected. When GC encounters errors or cannot make progress (for example, if there are no objects eligible for deletion), this backoff interval prevents excessive retries.

Increase this value to reduce system load when GC cannot make progress. Decrease it if you need faster retry attempts after transient errors.

Introduced in v25.3.3

The interval, in milliseconds, between invocations of the L0 (level-zero) garbage collection work loop when progress is being made.

L0 objects are short-term data objects in Tiered Storage associated with global epochs. This property controls how frequently GC runs when it successfully deletes objects. Lower values increase GC frequency, which can help maintain lower object counts but may increase S3 API usage.

Decrease this value if L0 object counts are growing too quickly and you need more aggressive garbage collection. Increase it to reduce S3 API costs in clusters with lower ingestion rates.

Introduced in v25.3.3

The minimum age, in milliseconds, of an L0 (level-zero) object before it becomes eligible for garbage collection.

This grace period delays deletion of L0 objects even after they become eligible based on epoch. The delay provides a safety buffer that can support recovery in cases involving accidental deletion or other operational issues.

Increase this value to extend the retention window for L0 objects, providing more time for recovery from operational errors. Decrease it to free up object storage space more quickly, but with less protection against accidental deletion.

Introduced in v26.1.1

The part size in bytes used for multipart uploads. The minimum of 5 MiB is the smallest non-terminal part size allowed by cloud object storage providers.

Cluster identifier.

Size (in bytes) for each compacted log segment.

Target backlog size for compaction controller. If not set the max backlog size is configured to 80% of total disk space available.

Derivative coefficient for compaction PID controller.

Integral coefficient for compaction PID controller.

Maximum number of I/O and CPU shares that compaction process can use.

Minimum number of I/O and CPU shares that compaction process can use.

Proportional coefficient for compaction PID controller. This must be negative, because the compaction backlog should decrease when the number of compaction shares increases.

The interval (in milliseconds) for updating the controller responsible for compaction tasks. The controller uses this interval to decide how to prioritize background compaction work, which is essential for maintaining efficient storage use.

This is an internal-only configuration and should be enabled only after consulting with Redpanda support.

How often Redpanda runs the collection loop when enable_consumer_group_metrics is set to consumer_lag. Updates will not be more frequent than health_monitor_max_metadata_age.

How often to run the collection loop when enable_consumer_group_metrics contains consumer_lag.

Reducing the value of consumer_group_lag_collection_interval_sec increases the metric collection frequency, which may raise resource utilization. In most environments, this impact is minimal, but it’s best practice to monitor broker resource usage in high-scale settings.

This property lets you enable the batch cache for the consumer offsets topic. By default, the cache for consumer offsets topic is disabled. Changing this property is not recommended in production systems, as it may affect performance. The change is applied only after the restart.

Interval between iterations of controller backend housekeeping loop.

The maximum number of cluster updates the controller can process at the same time. Higher values speed up cluster changes but use more resources.

Maximum capacity of rate limit accumulation in controller ACLs and users operations limit.

Maximum capacity of rate limit accumulation in controller configuration operations limit.

Maximum capacity of rate limit accumulation in controller move operations limit.

Maximum capacity of rate limit accumulation in controller node management operations limit.

Maximum capacity of rate limit accumulation in controller topic operations limit.

Maximum amount of time before Redpanda attempts to create a controller snapshot after a new controller command appears.

If set to true, move partitions between cores in runtime to maintain balanced partition distribution.

Interval, in milliseconds, between trigger and invocation of core balancing.

Unit: milliseconds

If set to true, and if after a restart the number of cores changes, Redpanda will move partitions between cores to maintain balanced partition distribution.

Enables CPU profiling for Redpanda.

The sample period for the CPU profiler.

The maximum size for a deployable WebAssembly binary that the broker can store.

The commit interval at which data transforms progress.

Enables WebAssembly-powered data transforms directly in the broker. When data_transforms_enabled is set to true, Redpanda reserves memory for data transforms, even if no transform functions are currently deployed. This memory reservation ensures that adequate resources are available for transform functions when they are needed, but it also means that some memory is allocated regardless of usage.

Buffer capacity for transform logs, per shard. Buffer occupancy is calculated as the total size of buffered log messages; that is, logs emitted but not yet produced.

Flush interval for transform logs. When a timer expires, pending logs are collected and published to the transform_logs topic.

Transform log lines truncate to this length. Truncation occurs after any character escaping.

The amount of memory to reserve per core for data transform (Wasm) virtual machines. Memory is reserved on boot. The maximum number of functions that can be deployed to a cluster is equal to data_transforms_per_core_memory_reservation / data_transforms_per_function_memory_limit.

The amount of memory to give an instance of a data transform (Wasm) virtual machine. The maximum number of functions that can be deployed to a cluster is equal to data_transforms_per_core_memory_reservation / data_transforms_per_function_memory_limit.

The percentage of available memory in the transform subsystem to use for read buffers.

The maximum amount of runtime to start up a data transform, and the time it takes for a single record to be transformed.

The percentage of available memory in the transform subsystem to use for write buffers.

Maximum amount of time the coordinator waits to snapshot after a command appears in the log.

Option to explicitly disable enforcement of datalake disk space usage.

The datalake disk usage monitor reclaims the overage multiplied by this this coefficient to compensate for data that is written during the idle period between control loop invocations.

Size, in bytes, of each memory block reserved for record translation, as tracked by the datalake scheduler.

The size, in bytes, of the block of disk reservation that the datalake manager will assign to each datalake scheduler when it runs out of local reservation.

The maximum number of translations that the datalake scheduler will allow to run at a given time. If a translation is requested, but the number of running translations exceeds this value, the request will be put to sleep temporarily, polling until capacity becomes available.

Time, in milliseconds, for a datalake translation as scheduled by the datalake scheduler. After a translation is scheduled, it will run until either the time specified has elapsed or all pending records on its source partition have been translated.

Size, in bytes, of the amount of scratch space datalake should use.

Size of the scratch space datalake soft limit expressed as a percentage of the datalake_scratch_space_size_bytes configuration value.

Size, in bytes, of the amount of per translator data that may be flushed to disk before the translator will upload and remove its current on disk data.

If set, how long debug bundles are kept in the debug bundle storage directory after they are created. If not set, debug bundles are kept indefinitely.

Path to the debug bundle storage directory. Note: Changing this path does not clean up existing debug bundles. If not set, the debug bundle is stored in the Redpanda data directory specified in the redpanda.yaml broker configuration file.

The recursion depth after which debug logging is enabled automatically for the log reader.