Cluster Configuration Properties
Cluster configuration properties are the same for all nodes in a cluster. They can be set at the cluster level.
For information on how to edit cluster properties, see Configuring Cluster Properties. Some properties require the cluster to be restarted in order to be applied; see a specific property’s reference for whether restart is required.
Object storage properties
cloud_storage_azure_storage_account
The name of the Azure storage account to use with Tiered Storage. If null
, the property is disabled.
Type: string
Default: null
Restart required: yes
Supported versions: Redpanda v23.1 or later
cloud_storage_azure_container
The name of the Azure container to use with Tiered Storage. If null
, the property is disabled.
The container must belong to cloud_storage_azure_storage_account. |
Type: string
Default: null
Restart required: yes
Supported versions: Redpanda v23.1 or later
cloud_storage_azure_shared_key
The shared key to be used for Azure Shared Key authentication with the Azure storage account configured by cloud_storage_azure_storage_account. If null
, the property is disabled.
Redpanda expects this key string to be Base64 encoded. |
Type: string
Default: null
Restart required: yes
Supported versions: Redpanda v23.1 or later
cloud_storage_bucket
AWS or GCS bucket that should be used to store data.
Default: null
Restart required: yes
cloud_storage_cache_size
Max size of object storage cache.
Units: bytes
Default: 21474836480 (20 GiB)
Restart required: yes
cloud_storage_credentials_source
The source of credentials used to connect to cloud services.
Default: config_file
Valid values: config_file, aws_instance_metadata, sts, gcp_instance_metadata
Restart required: yes
cloud_storage_disable_tls
Disable TLS for all S3 or GCS connections.
Type: boolean
Default: false
Restart required: yes
cloud_storage_enabled
Enable object storage. Must be set to true
to use Tiered Storage or Remote Read Replicas.
Type: boolean
Default: false
Restart required: yes
cloud_storage_max_connections
Max number of simultaneous connections to S3 per shard. Includes connections used for both uploads and downloads.
Units: number of simultaneous connections
Default: 20
Restart required: yes
Cluster management properties
enable_auto_rebalance_on_node_add
Enable automatic partition rebalancing when new nodes are added.
Type: boolean
Default: false
Restart required: no
enable_controller_log_rate_limiting
Flag to enable limiting the write rate for the controller log.
Type: boolean
Default: false
Restart required: no
enable_leader_balancer
Enable automatic leadership rebalancing. Mode is set by leader_balancer_mode
.
Type: boolean
Default: true
Restart required: no
enable_rack_awareness
Enable rack-aware replica assignment.
Type: boolean
Default: false
Restart required: no
leader_balancer_mode
Mode of the leader balancer for optimizing movements of leadership between shards (logical CPU cores). Enabled by enable_leader_balancer
.
Valid modes:
-
random_hill_climbing
: a shard is randomly chosen and leadership is moved to it if the load on the original shard is reduced. -
greedy_balanced_shards
: leadership movement is based on a greedy heuristic of moving leaders from the most loaded shard to the least loaded shard.
Default: random_hill_climbing
Restart required: no
Supported versions: Redpanda v23.1 or later
partition_autobalancing_mode
Mode of partition balancing for a cluster.
Available modes:
-
node_add
: partition balancing happens when a node is added. -
continuous
: partition balancing happens automatically to maintain optimal performance and availability, based on continuous monitoring for node changes (same asnode_add
) and also high disk usage. This option requires an Enterprise license, and it is customized by partition_autobalancing_node_availability_timeout_sec and partition_autobalancing_max_disk_usage_percent properties. -
off
: partition balancing is disabled. This option is not recommended for production clusters.
Default: node_add
Restart required: no
Related topics:
partition_autobalancing_node_availability_timeout_sec
This property applies only when partition_autobalancing_mode is set to continuous .
|
When a node is unavailable for at least this timeout duration, it triggers Redpanda to move partitions off of the node.
Units: seconds
Default: 900 (15 min)
Restart required: no
Related topics:
partition_autobalancing_max_disk_usage_percent
This property applies only when partition_autobalancing_mode is set to continuous .
|
When the disk usage of a node exceeds this threshold, it triggers Redpanda to move partitions off of the node.
Units: percent of disk used
Default: 80
Range: [5, 100]
Related topics:
Kafka API properties
kafka_admin_topic_api_rate
Target quota rate for partition mutations per default_window_sec
. If null
, the property is disabled, and no quota rate is applied.
Units: partition mutations per default_window_second
Default: null
Range: [1, …]
Restart required: no
Related properties:
kafka_client_group_byte_rate_quota
A map specifying the produce-rate quota per client group.
The configurable fields:
-
group_name
: name of a client group -
clients_prefix
: prefix to prepend to the name of each client belonging to the group specified bygroup_name
-
quota
: produce-rate quota of each client in bytes per second
An example: ([{'group_name': 'first_group','clients_prefix': 'group_1','quota': 10240}])
Default: {} (empty map)
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_client_group_fetch_byte_rate_quota
A map specifying the fetch-rate quota per client group.
The configurable fields:
-
group_name
: name of a client group -
clients_prefix
: prefix to prepend to the name of each client belonging to the group specified bygroup_name
-
quota
: fetch-rate quota of each client in bytes per second
An example: ([{'group_name': 'first_group','clients_prefix': 'group_1','quota': 10240}])
Default: {} (empty map)
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
enable_sasl
Enable SASL authentication for Kafka connections.
Type: boolean
Default: false
Restart required: no
enable_schema_id_validation
Mode to enable server-side schema ID validation.
Valid values:
-
none
: schema validation is disabled (no schema ID checks are done). Associated topic properties cannot be modified. -
redpanda
: schema validation is enabled. Only Redpanda topic properties are accepted. -
compat
: schema validation is enabled. Both Redpanda and compatible topic properties are accepted.
Default: none
Restart required: no
Related topics:
fetch_max_bytes
Maximum number of bytes returned in a fetch request.
Units: bytes
Default: 57671680 (55 MiB)
Restart required: no
group_max_session_timeout_ms
The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures.
Units: milliseconds
Default: 300000 (300 sec)
Restart required: no
group_min_session_timeout_ms
The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating which can overwhelm broker resources.
Units: milliseconds
Default: 6000 (6 sec)
Restart required: no
kafka_connection_rate_limit
Maximum connections per second for one core. If null
(the default), the number of connections per second is unlimited.
Units: number of connections per second, per core
Default: null
Range: [1, …]
Restart required: yes
Related topics:
kafka_connection_rate_limit_overrides
Overrides the maximum connections per second for one core for the specified IP addresses (for example, ['127.0.0.1:90', '50.20.1.1:40']
)
Type: string
Default: null
Restart required: no
Related topics:
kafka_connections_max
Maximum number of Kafka client connections per broker. If null
, the property is disabled.
Units: number of Kafka client connections per broker
Default: null
Restart required: no
Related topics:
kafka_connections_max_overrides
A list of IP addresses for which Kafka client connection limits are overridden and don’t apply. For example, (['127.0.0.1:90', '50.20.1.1:40']).
Default: {} (empty list)
Restart required: no
Related topics:
kafka_connections_max_per_ip
Maximum number of Kafka client connections per IP address, per broker. If null
, the property is disabled.
Units: number of Kafka client connections per IP address, per broker
Default: null
Restart required: no
Related topics:
kafka_enable_authorization
Flag to require authorization for Kafka connections. If null
, the property is disabled, and authorization is instead enabled by enable_sasl.
Valid values:
-
null
: Ignored. Authorization is enabled withenable_sasl
:true
-
true
: authorization is required. -
false
: authorization is disabled.
Type: boolean
Default: null
Related properties:
-
kafka_api[].authentication_method
kafka_enable_partition_reassignment
Enable the Kafka partition reassignment API.
Type: boolean
Default: true
Restart required: no
Supported versions: Redpanda v23.1 or later
kafka_group_recovery_timeout_ms
Kafka group recovery timeout.
Units: milliseconds
Default: 30000 (30 sec)
Restart required: no
kafka_mtls_principal_mapping_rules
Principal mapping rules for mTLS authentication on the Kafka API. If null
, the property is disabled.
Default: null
Restart required: no
kafka_nodelete_topics
A list of topics that are protected from deletion and configuration changes by Kafka clients. Set by default to a list of Redpanda internal topics.
Default: ['audit', 'consumer_offsets', '_schemas']
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_noproduce_topics
A list of topics that are protected from being produced to by Kafka clients. Set by default to a list of Redpanda internal topics.
Default: ['__audit']
Restart required: no
Supported versions: Redpanda v23.1 or later
kafka_qdc_enable
Enable Kafka queue depth control.
Type: boolean
Default: false
Restart required: yes
kafka_qdc_max_latency_ms
Maximum latency threshold for Kafka queue depth control depth tracking.
Units: milliseconds
Default: 80
Restart required: yes
kafka_quota_balancer_node_period_ms
The period at which the intra-node throughput quota balancer runs.
It may take longer for the balancer to complete a single balancing step than the period this property specifies, so the actual period may be more than configured here.
If 0
, the balancer is disabled and all throughput quotas are immutable.
Units: milliseconds
Default: 750
Range: [0, ]
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_quota_balancer_min_shard_throughput_ratio
The minimum value of the throughput quota a shard can get in the process of quota balancing, expressed as a ratio of default shard quota. While the value applies equally to ingress and egress traffic, the default shard quota can be different for ingress and egress and therefore result in different minimum throughput bytes-per-second (bps) values.
Both kafka_quota_balancer_min_shard_throughput_ratio
and kafka_quota_balancer_min_shard_throughput_bps can be specified at the same time. In this case, the balancer will not decrease the effective shard quota below the largest bps value of each of these two properties.
If set to 0.0
, the minimum is disabled. If set to 1.0
, then the balancer won’t be able to rebalance quota without violating this ratio, consequently precluding the balancer from adjusting shards' quotas.
Type: double
Units: ratio of default shard quota
Default: 0.01
Range: [0.0, 1.0]
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_quota_balancer_min_shard_throughput_bps
The minimum value of the throughput quota a shard can get in the process of quota balancing, expressed in bytes per second. The value applies equally to ingress and egress traffic.
kafka_quota_balancer_min_shard_throughput_bps doesn’t override the limit settings, kafka_throughput_limit_node_in_bps and kafka_throughput_limit_node_out_bps. Consequently, the value of
kafka_throughput_limit_node_in_bps
or kafka_throughput_limit_node_out_bps
can result in lesser throughput than kafka_quota_balancer_min_shard_throughput_bps.
Both kafka_quota_balancer_min_shard_throughput_ratio and kafka_quota_balancer_min_shard_throughput_bps can be specified at the same time. In this case, the balancer will not decrease the effective shard quota below the largest bps value of each of these two properties.
If set to 0
, no minimum is enforced.
Units: bytes per second
Default: 256
Range: [0, …]
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_quota_balancer_window_ms
Time window used to average the current throughput measurement for the quota balancer.
Units: milliseconds
Default: 5000
Range: [1, …]
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_rpc_server_tcp_recv_buf
Size of the Kafka server TCP receive buffer. If null
, the property is disabled.
Units: bytes
Default: null
Range: [32 KiB, …], aligned to 4096 bytes
kafka_rpc_server_tcp_send_buf
Size of the Kafka server TCP transmit buffer. If null
, the property is disabled.
Units: bytes
Default: null
Range: [32 KiB, …], aligned to 4096 bytes
kafka_throughput_limit_node_in_bps
The maximum rate of all ingress Kafka API traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.).
If null
, the property is disabled, and traffic is not limited.
Units: bytes per second
Default: null
Range: [1, …]
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
kafka_throughput_limit_node_out_bps
The maximum rate of all egress Kafka traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.).
If null
, the property is disabled, and traffic is not limited.
Units: bytes per second
Default: null
Range: [1, …]
Restart required: no
Related topics:
Supported versions: Redpanda v23.1 or later
log_segment_ms
Default lifetime of log segments. If null
, the property is disabled, and no default lifetime is set. This property can also be set in the Kafka API using the Kafka-compatible alias, log.roll.ms
.
The topic property segment.ms
overrides the value of log_segment_ms
at the topic level.
Units: milliseconds
Default: null
Range: [60000 (60 sec), …]
Restart required: no
Related properties:
Supported versions: Redpanda v23.1 or later
rm_sync_timeout_ms
Resource manager’s synchronization timeout. Maximum time for this node to wait for internal state machine to catch up with all events written by previous leaders before rejecting a request.
Units: milliseconds
Default: 10000 (10 sec)
Restart required: yes
rpc_server_listen_backlog
Maximum TCP connection queue length for Kafka server and internal RPC server. If null
(the default value), no queue length is set.
Units: number of queue entries
Default: null
Range: [1, …]
Restart required: yes
rpc_server_tcp_recv_buf
Internal RPC TCP receive buffer size. If null
(the default value), no buffer size is set by Redpanda.
Units: bytes
Default: null
Range: [32 KiB, …], aligned to 4096 bytes
Restart required: yes
rpc_server_tcp_send_buf
Internal RPC TCP send buffer size. If null
(the default value), no buffer size is set by Redpanda.
Units: bytes
Default: null
Range: [32 KiB, …], aligned to 4096 bytes
Restart required: yes
sasl_kerberos_config
The location of the Kerberos krb5.conf
file for Redpanda.
Type: string
Default: /etc/krb5.conf
Restart required: no
sasl_kerberos_keytab
The location of the Kerberos keytab file for Redpanda.
Type: string
Default: /var/lib/redpanda/redpanda.keytab
Restart required: no
sasl_kerberos_principal
The primary of the Kerberos Service Principal Name (SPN) for Redpanda.
Type: string
Default: redpanda
Restart required: no
sasl_kerberos_principal_mapping
Rules for mapping Kerberos principal names to Redpanda user principals.
Type: array of string
Default: ["DEFAULT"]
Restart required: no
sasl_mechanisms
A list of supported SASL mechanisms. SCRAM
and GSSAPI
are allowed.
Type: array of string
Default: ["SCRAM"]
Valid values: "SCRAM"
, "GSSAPI"
Restart required: no
target_quota_byte_rate
Target quota byte rate.
The target_quota_byte_rate
property applies to a producer client that isn’t a member of a client group configured by kafka_client_group_byte_rate_quota
. It sets the maximum throughput quota of a client sending to a Redpanda broker node.
Units: bytes per second
Default: 2147483648 (2 GiB)
Range: [1048576 (1 MiB), …]
Restart required: no
Related topics:
Metrics properties
aggregate_metrics
Enable aggregation of metrics returned by the /metrics endpoint. Metric aggregation is performed by summing the values of samples by labels and is done when it makes sense by the shard and/or partition labels.
Type: boolean
Default: false
Restart required: yes
disable_metrics
Disable registering metrics exposed on the internal metrics endpoint.
Type: boolean
Default: false
Restart required: yes
disable_public_metrics
Disable registering metrics exposed on the public metrics endpoint.
Type: boolean
Default: false
Restart required: yes
enable_metrics_reporter
Enable the cluster metrics reporter. If true
, the metrics reporter collects and exports to Redpanda Data a set of customer usage metrics at the interval set by metrics_reporter_report_interval.
The cluster metrics of the metrics reporter are different from monitoring metrics.
|
Type: boolean
Default: true
Restart required: no
Raft properties
raft_learner_recovery_rate
Raft learner recovery rate limit. Throttles the rate of data communicated to nodes (learners) that need to catch up to leaders.
This rate limit is placed on a node sending data to a recovering node. Each sending node will be limited to this rate. The recovering node will accept data as fast as possible according to the combined limits of all healthy nodes in the cluster. For example, if two nodes are sending data to the recovering node, and raft_learner_recovery_rate
is 100 MB/sec, then the recovering node will recover at a rate of 200 MB/sec.
Units: bytes per second
Default: 104857600 (100 MB/sec)
Restart required: no
Storage properties
cloud_storage_cache_size_percent
Maximum size of the cloud cache.
The property cloud_storage_cache_size
controls the same limit expressed as a fixed number of bytes.
cloud_storage_cache_size_percent overrides cloud_storage_cache_size .
|
Units: percentage of total disk size
Default: 10
Restart required: no
delete_retention_ms
Delete segments that are older than this age.
The topic property retention.ms
overrides the value of delete_retention_ms
at the topic level.
The
|
Units: milliseconds
Default: 604800000 (1 week)
Restart required: no
disk_reservation_percent
Amount of disk space to reserve for general system overhead.
Units: percentage of total disk size
Default: 20
Restart required: no
log_cleanup_policy
Default cleanup policy for topic logs.
The topic property cleanup.policy
overrides the value of log_cleanup_policy
at the topic level.
Default: delete
Valid Values: compact
, delete
, compact,delete
, none
Restart required: no
log_compaction_interval_ms
How often to trigger background compaction.
Units: milliseconds
Default: 10000 (10 sec)
Restart required: no
log_compression_type
Default topic compression type (gzip, snappy, lz4, zstd, producer, or none).
The topic property compression.type
overrides the value of log_compression_type
at the topic level.
Default: producer
Valid values: gzip
, snappy
, lz4
, zstd
, producer
, none
Restart required: no
log_message_timestamp_type
Default timestamp type for topic messages (CreateTime or LogAppendTime).
The topic property message.timestamp.type
overrides the value of log_message_timestamp_type
at the topic level.
Default: CreateTime
Valid values: CreateTime
, LogAppendTime
Restart required: no
retention_local_target_capacity_bytes
The target capacity in bytes that log storage will try to use before additional retention rules will take over to trim data in order to meet the target. When no target is specified, storage usage is unbounded.
Redpanda Data recommends setting only one of retention_local_target_capacity_bytes or retention_local_target_capacity_percent . If both are set, the minimum of the two is used as the effective target capacity.
|
Units: percentage of total disk size
Default: null
Restart required: no
retention_local_target_capacity_percent
The target capacity in percent of unreserved space (disk_reservation_percent
) that log storage will try to use before additional retention rules will take over to trim data in order to meet the target. When no target is specified storage usage is unbounded.
Redpanda Data recommends setting only one of retention_local_target_capacity_bytes or retention_local_target_capacity_percent . If both are set, the minimum of the two is used as the effective target capacity.
|
Units: percentage of total disk size
Default: remaining percentage of disk space after accounting for disk_reservation_percent
and cloud_storage_cache_size_percent
Restart required: no
retention_local_strict
Flag to allow Tiered Storage topics to expand to consumable retention policy limits.
When this flag is enabled, non-local retention settings are used, and local retention settings are used to inform data removal policies in low-disk space scenarios.
Type: boolean
Default: false
Restart required: no
retention_local_target_bytes_default
Local retention size target for partitions of topics with object storage write enabled. If null
, the property is disabled.
This property can be overridden on a per-topic basis by setting retention.local.target.bytes
in each topic enabled for Tiered Storage. See Configure message retention.
:::note
Both retention_local_target_bytes_default
and retention_local_target_ms_default
can be set. The limit that is reached earlier is applied.
:::
Units: bytes
Default: null
Restart required: no
Related properties:
retention_local_target_ms_default
Local retention time target for partitions of topics with object storage write enabled.
This property can be overridden on a per-topic basis by setting retention.local.target.ms
in each topic enabled for Tiered Storage. See Configure message retention.
Both retention_local_target_bytes_default and retention_local_target_ms_default can be set. The limit that is reached earlier is applied.
|
Units: milliseconds
Default: 86400000 (24 hours)
Restart required: no
Related properties:
retention_local_trim_interval
The period that disk usage is checked for disk pressure, and data is optionally trimmed to meet the target.
Units: seconds
Default: 30
Restart required: no
space_management_enable
Flag to enable automatic disk space management.
Type: boolean
Default: true
for new clusters, false
for upgraded clusters
Restart required: no
storage_strict_data_init
Requires that an empty file named .redpanda_data_dir
be present in the data directory. If set true
, Redpanda will refuse to start if the file is not found in the data directory.
Type: boolean
Default: false
Restart required: no
storage_ignore_timestamps_in_future_sec
The maximum number of seconds that a record’s timestamp can be ahead of a Redpanda broker’s clock and still be used when deciding whether to clean up the record for data retention. This property makes possible the timely cleanup of records from clients with clocks that are drastically unsynchronized relative to Redpanda.
When determining whether to clean up a record with timestamp more than storage_ignore_timestamps_in_future_sec
seconds ahead of the broker, Redpanda ignores the record’s timestamp and instead uses a valid timestamp of another record in the same segment, or (if another record’s valid timestamp is unavailable) the timestamp of when the segment file was last modified (mtime).
By default, storage_ignore_timestamps_in_future_sec
is disabled (null).
To figure out whether to set
|
Units: seconds
Default: null
Restart required: no
Supported versions: Redpanda v22.3 or later
Support properties
legacy_permit_unsafe_log_operations
Flag enabling a Redpanda cluster operator to use unsafe control characters within strings such as consumer group names or user names.
This flag applies only for Redpanda clusters v23.1 or earlier that have upgraded to v23.2 or later. Newly-created Redpanda clusters v23.2 or later ignore this property.
Type: boolean
Default: true
Restart required: no
Related properties: legacy_unsafe_log_warning_interval_sec
legacy_unsafe_log_warning_interval_sec
Period at which to log a warning about using unsafe strings containing control characters.
If unsafe strings are permitted by legacy_permit_unsafe_log_operations
, a warning will be logged at an interval specified by this property.
Units: seconds
Default: 300
Restart required: no
Related properties: legacy_permit_unsafe_log_operations
Topic and partition properties
auto_create_topics_enabled
Allow automatic topic creation.
Type: boolean
Default: false
Restart required: no
default_topic_partitions
Default number of partitions per topic.
Units: number of partitions per topic
Default: 1
Restart required: no
default_topic_replications
Default replication factor for new topics.
The topic property replication.factor
overrides the value of default_topic_replications
at the topic level.
Units: number of replicas per topic
Default: 1
Range: [1, …], must be odd
Restart required: no
internal_topic_replication_factor
Target replication factor for internal topics.
Units: number of replicas per topic
Default: 3
Restart required: yes
retention_bytes
Default maximum number of bytes per partition on disk before triggering deletion of the oldest messages. If null
(the default value), no limit is applied.
The topic property retention.bytes
overrides the value of retention_bytes
at the topic level.
Units: bytes per partition
Default: null
Restart required: no
Transaction properties
enable_transactions
Enable transactions (atomic writes).
Type: boolean
Default: true
Restart required: yes
seq_table_min_size
The minimum threshold number of sessions to keep in the seq table. Not affected by compaction.
Default: 1000
Restart required: yes
tm_sync_timeout_ms
Transaction manager’s synchronization timeout. Maximum time to wait for internal state machine to catch up before rejecting a request.
Units: milliseconds
Default: 10000 (10 sec)
Restart required: yes
tm_violation_recovery_policy
Describes how to recover from an invariant violation at the transaction coordinator level.
Default: crash
Restart required: yes
transaction_coordinator_cleanup_policy
Cleanup policy for a transaction coordinator topic.
Default: delete
Valid Values: compact
, delete
, compact,delete
, none
Restart required: no
transaction_coordinator_delete_retention_ms
Delete segments older than this age. To ensure transaction state is retained as long as the longest-running transaction, make sure this is no less than transactional_id_expiration_ms.
Units: milliseconds
Default: 604800000 (1 week)
Restart required: no