Cluster Configuration Properties

Whether admin API clients must provide HTTP basic authentication headers.

Default: false

Restart required: no

List of superuser usernames.

Default: null

Restart required: no

The name of the Azure storage account to use with Tiered Storage. If null, the property is disabled.

Type: string

Default: null

Restart required: yes

Supported versions: Redpanda v23.1 or later

The name of the Azure container to use with Tiered Storage. If null, the property is disabled.

Type: string

Default: null

Restart required: yes

Supported versions: Redpanda v23.1 or later

The shared key to be used for Azure Shared Key authentication with the Azure storage account configured by cloud_storage_azure_storage_account. If null, the property is disabled.

Type: string

Default: null

Restart required: yes

Supported versions: Redpanda v23.1 or later

AWS or GCP access key.

Default: null

Restart required: yes

Optional API endpoint.

Default: null

Restart required: yes

TLS port override.

Default: 443

Restart required: yes

AWS or GCS bucket that should be used to store data.

Default: null

Restart required: yes

Max size of object storage cache.

Units: bytes

Default: 21474836480 (20 GiB)

Restart required: yes

The source of credentials used to connect to cloud services.

Default: config_file

Valid values: config_file, aws_instance_metadata, sts, gcp_instance_metadata

Restart required: yes

Disable TLS for all S3 or GCS connections.

Type: boolean

Default: false

Restart required: yes

Enable object storage. Must be set to true to use Tiered Storage or Remote Read Replicas.

Type: boolean

Default: false

Restart required: yes

Max number of simultaneous connections to S3 per shard. Includes connections used for both uploads and downloads.

Units: number of simultaneous connections

Default: 20

Restart required: yes

AWS or GCP region that houses the bucket used for storage.

Type: string

Default: null

Restart required: yes

AWS or GCP secret key.

Type: string

Default: null

Restart required: yes

Path to certificate that should be used to validate server certificate during TLS handshake.

Type: string

Default: null

Restart required: yes

Cluster identifier.

Type: string

Default: null

Restart required: no

Enable automatic partition rebalancing when new nodes are added.

Type: boolean

Default: false

Restart required: no

Flag to enable limiting the write rate for the controller log.

Type: boolean

Default: false

Restart required: no

Enable automatic leadership rebalancing. Mode is set by leader_balancer_mode.

Type: boolean

Default: true

Restart required: no

Enable rack-aware replica assignment.

Type: boolean

Default: false

Restart required: no

Mode of the leader balancer for optimizing movements of leadership between shards (logical CPU cores). Enabled by enable_leader_balancer.

Valid modes:

Default: random_hill_climbing

Restart required: no

Supported versions: Redpanda v23.1 or later

Mode of partition balancing for a cluster.

Available modes:

Default: node_add

Restart required: no

Related topics:

When a node is unavailable for at least this timeout duration, it triggers Redpanda to move partitions off of the node.

Units: seconds

Default: 900 (15 min)

Restart required: no

Related topics:

When the disk usage of a node exceeds this threshold, it triggers Redpanda to move partitions off of the node.

Units: percent of disk used

Default: 80

Range: [5, 100]

Related topics:

Target quota rate for partition mutations per default_window_sec. If null, the property is disabled, and no quota rate is applied.

Units: partition mutations per default_window_second

Default: null

Range: [1, …​]

Restart required: no

Related properties:

A map specifying the produce-rate quota per client group.

The configurable fields:

An example: ([{'group_name': 'first_group','clients_prefix': 'group_1','quota': 10240}])

Default: {} (empty map)

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

A map specifying the fetch-rate quota per client group.

The configurable fields:

An example: ([{'group_name': 'first_group','clients_prefix': 'group_1','quota': 10240}])

Default: {} (empty map)

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

Enable idempotent producers.

Type: boolean

Default: true

Restart required: yes

Enable SASL authentication for Kafka connections.

Type: boolean

Default: false

Restart required: no

Mode to enable server-side schema ID validation.

Valid values:

Default: none

Restart required: no

Related topics:

Maximum number of bytes returned in a fetch request.

Units: bytes

Default: 57671680 (55 MiB)

Restart required: no

The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures.

Units: milliseconds

Default: 300000 (300 sec)

Restart required: no

The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating which can overwhelm broker resources.

Units: milliseconds

Default: 6000 (6 sec)

Restart required: no

Maximum connections per second for one core. If null (the default), the number of connections per second is unlimited.

Units: number of connections per second, per core

Default: null

Range: [1, …​]

Restart required: yes

Related topics:

Overrides the maximum connections per second for one core for the specified IP addresses (for example, ['127.0.0.1:90', '50.20.1.1:40'])

Type: string

Default: null

Restart required: no

Related topics:

Maximum number of Kafka client connections per broker. If null, the property is disabled.

Units: number of Kafka client connections per broker

Default: null

Restart required: no

Related topics:

A list of IP addresses for which Kafka client connection limits are overridden and don’t apply. For example, (['127.0.0.1:90', '50.20.1.1:40']).

Default: {} (empty list)

Restart required: no

Related topics:

Maximum number of Kafka client connections per IP address, per broker. If null, the property is disabled.

Units: number of Kafka client connections per IP address, per broker

Default: null

Restart required: no

Related topics:

Flag to require authorization for Kafka connections. If null, the property is disabled, and authorization is instead enabled by enable_sasl.

Valid values:

Type: boolean

Default: null

Related properties:

Enable the Kafka partition reassignment API.

Type: boolean

Default: true

Restart required: no

Supported versions: Redpanda v23.1 or later

Kafka group recovery timeout.

Units: milliseconds

Default: 30000 (30 sec)

Restart required: no

Principal mapping rules for mTLS authentication on the Kafka API. If null, the property is disabled.

Default: null

Restart required: no

A list of topics that are protected from deletion and configuration changes by Kafka clients. Set by default to a list of Redpanda internal topics.

Default: ['audit', 'consumer_offsets', '_schemas']

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

A list of topics that are protected from being produced to by Kafka clients. Set by default to a list of Redpanda internal topics.

Default: ['__audit']

Restart required: no

Supported versions: Redpanda v23.1 or later

Enable Kafka queue depth control.

Type: boolean

Default: false

Restart required: yes

Maximum latency threshold for Kafka queue depth control depth tracking.

Units: milliseconds

Default: 80

Restart required: yes

The period at which the intra-node throughput quota balancer runs.

It may take longer for the balancer to complete a single balancing step than the period this property specifies, so the actual period may be more than configured here.

If 0, the balancer is disabled and all throughput quotas are immutable.

Units: milliseconds

Default: 750

Range: [0, ]

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

The minimum value of the throughput quota a shard can get in the process of quota balancing, expressed as a ratio of default shard quota. While the value applies equally to ingress and egress traffic, the default shard quota can be different for ingress and egress and therefore result in different minimum throughput bytes-per-second (bps) values.

Both kafka_quota_balancer_min_shard_throughput_ratio and kafka_quota_balancer_min_shard_throughput_bps can be specified at the same time. In this case, the balancer will not decrease the effective shard quota below the largest bps value of each of these two properties.

If set to 0.0, the minimum is disabled. If set to 1.0, then the balancer won’t be able to rebalance quota without violating this ratio, consequently precluding the balancer from adjusting shards' quotas.

Type: double

Units: ratio of default shard quota

Default: 0.01

Range: [0.0, 1.0]

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

The minimum value of the throughput quota a shard can get in the process of quota balancing, expressed in bytes per second. The value applies equally to ingress and egress traffic.

kafka_quota_balancer_min_shard_throughput_bps doesn’t override the limit settings, kafka_throughput_limit_node_in_bps and kafka_throughput_limit_node_out_bps. Consequently, the value of kafka_throughput_limit_node_in_bps or kafka_throughput_limit_node_out_bps can result in lesser throughput than kafka_quota_balancer_min_shard_throughput_bps.

Both kafka_quota_balancer_min_shard_throughput_ratio and kafka_quota_balancer_min_shard_throughput_bps can be specified at the same time. In this case, the balancer will not decrease the effective shard quota below the largest bps value of each of these two properties.

If set to 0, no minimum is enforced.

Units: bytes per second

Default: 256

Range: [0, …​]

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

Time window used to average the current throughput measurement for the quota balancer.

Units: milliseconds

Default: 5000

Range: [1, …​]

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

Size of the Kafka server TCP receive buffer. If null, the property is disabled.

Units: bytes

Default: null

Range: [32 KiB, …​], aligned to 4096 bytes

Size of the Kafka server TCP transmit buffer. If null, the property is disabled.

Units: bytes

Default: null

Range: [32 KiB, …​], aligned to 4096 bytes

The maximum rate of all ingress Kafka API traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.).

If null, the property is disabled, and traffic is not limited.

Units: bytes per second

Default: null

Range: [1, …​]

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

The maximum rate of all egress Kafka traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.).

If null, the property is disabled, and traffic is not limited.

Units: bytes per second

Default: null

Range: [1, …​]

Restart required: no

Related topics:

Supported versions: Redpanda v23.1 or later

Default lifetime of log segments. If null, the property is disabled, and no default lifetime is set. This property can also be set in the Kafka API using the Kafka-compatible alias, log.roll.ms.

The topic property segment.ms overrides the value of log_segment_ms at the topic level.

Units: milliseconds

Default: null

Range: [60000 (60 sec), …​]

Restart required: no

Related properties:

Supported versions: Redpanda v23.1 or later

Resource manager’s synchronization timeout. Maximum time for this node to wait for internal state machine to catch up with all events written by previous leaders before rejecting a request.

Units: milliseconds

Default: 10000 (10 sec)

Restart required: yes

Maximum TCP connection queue length for Kafka server and internal RPC server. If null (the default value), no queue length is set.

Units: number of queue entries

Default: null

Range: [1, …​]

Restart required: yes

Internal RPC TCP receive buffer size. If null (the default value), no buffer size is set by Redpanda.

Units: bytes

Default: null

Range: [32 KiB, …​], aligned to 4096 bytes

Restart required: yes

Internal RPC TCP send buffer size. If null (the default value), no buffer size is set by Redpanda.

Units: bytes

Default: null

Range: [32 KiB, …​], aligned to 4096 bytes

Restart required: yes

The location of the Kerberos krb5.conf file for Redpanda.

Type: string

Default: /etc/krb5.conf

Restart required: no

The location of the Kerberos keytab file for Redpanda.

Type: string

Default: /var/lib/redpanda/redpanda.keytab

Restart required: no

The primary of the Kerberos Service Principal Name (SPN) for Redpanda.

Type: string

Default: redpanda

Restart required: no

Rules for mapping Kerberos principal names to Redpanda user principals.

Type: array of string

Default: ["DEFAULT"]

Restart required: no

A list of supported SASL mechanisms. SCRAM and GSSAPI are allowed.

Type: array of string

Default: ["SCRAM"]

Valid values: "SCRAM", "GSSAPI"

Restart required: no

Target quota byte rate.

The target_quota_byte_rate property applies to a producer client that isn’t a member of a client group configured by kafka_client_group_byte_rate_quota. It sets the maximum throughput quota of a client sending to a Redpanda broker node.

Units: bytes per second

Default: 2147483648 (2 GiB)

Range: [1048576 (1 MiB), …​]

Restart required: no

Related topics:

Target fetch-size quota byte rate. If null, the property is disabled, and no quota byte rate is applied.

Units: bytes per second

Default: null

Restart required: no

Supported versions: Redpanda v23.1 or later

Enable aggregation of metrics returned by the /metrics endpoint. Metric aggregation is performed by summing the values of samples by labels and is done when it makes sense by the shard and/or partition labels.

Type: boolean

Default: false

Restart required: yes

Disable registering metrics exposed on the internal metrics endpoint.

Type: boolean

Default: false

Restart required: yes

Disable registering metrics exposed on the public metrics endpoint.

Type: boolean

Default: false

Restart required: yes

Enable the cluster metrics reporter. If true, the metrics reporter collects and exports to Redpanda Data a set of customer usage metrics at the interval set by metrics_reporter_report_interval.

Type: boolean

Default: true

Restart required: no

Raft learner recovery rate limit. Throttles the rate of data communicated to nodes (learners) that need to catch up to leaders.

This rate limit is placed on a node sending data to a recovering node. Each sending node will be limited to this rate. The recovering node will accept data as fast as possible according to the combined limits of all healthy nodes in the cluster. For example, if two nodes are sending data to the recovering node, and raft_learner_recovery_rate is 100 MB/sec, then the recovering node will recover at a rate of 200 MB/sec.

Units: bytes per second

Default: 104857600 (100 MB/sec)

Restart required: no

Maximum size of the cloud cache.

The property cloud_storage_cache_size controls the same limit expressed as a fixed number of bytes.

Units: percentage of total disk size

Default: 10

Restart required: no

Delete segments that are older than this age.

The topic property retention.ms overrides the value of delete_retention_ms at the topic level.

Units: milliseconds

Default: 604800000 (1 week)

Restart required: no

Amount of disk space to reserve for general system overhead.

Units: percentage of total disk size

Default: 20

Restart required: no

Default cleanup policy for topic logs.

The topic property cleanup.policy overrides the value of log_cleanup_policy at the topic level.

Default: delete

Valid Values: compact, delete, compact,delete, none

Restart required: no

How often to trigger background compaction.

Units: milliseconds

Default: 10000 (10 sec)

Restart required: no

Default topic compression type (gzip, snappy, lz4, zstd, producer, or none).

The topic property compression.type overrides the value of log_compression_type at the topic level.

Default: producer

Valid values: gzip, snappy, lz4, zstd, producer, none

Restart required: no

Default timestamp type for topic messages (CreateTime or LogAppendTime).

The topic property message.timestamp.type overrides the value of log_message_timestamp_type at the topic level.

Default: CreateTime

Valid values: CreateTime, LogAppendTime

Restart required: no

The target capacity in bytes that log storage will try to use before additional retention rules will take over to trim data in order to meet the target. When no target is specified, storage usage is unbounded.

Units: percentage of total disk size

Default: null

Restart required: no

The target capacity in percent of unreserved space (disk_reservation_percent) that log storage will try to use before additional retention rules will take over to trim data in order to meet the target. When no target is specified storage usage is unbounded.

Units: percentage of total disk size

Default: remaining percentage of disk space after accounting for disk_reservation_percent and cloud_storage_cache_size_percent

Restart required: no

Flag to allow Tiered Storage topics to expand to consumable retention policy limits.

When this flag is enabled, non-local retention settings are used, and local retention settings are used to inform data removal policies in low-disk space scenarios.

Type: boolean

Default: false

Restart required: no

Local retention size target for partitions of topics with object storage write enabled. If null, the property is disabled.

This property can be overridden on a per-topic basis by setting retention.local.target.bytes in each topic enabled for Tiered Storage. See Configure message retention.

:::note Both retention_local_target_bytes_default and retention_local_target_ms_default can be set. The limit that is reached earlier is applied. :::

Units: bytes

Default: null

Restart required: no

Related properties:

Local retention time target for partitions of topics with object storage write enabled.

This property can be overridden on a per-topic basis by setting retention.local.target.ms in each topic enabled for Tiered Storage. See Configure message retention.

Units: milliseconds

Default: 86400000 (24 hours)

Restart required: no

Related properties:

The period that disk usage is checked for disk pressure, and data is optionally trimmed to meet the target.

Units: seconds

Default: 30

Restart required: no

Flag to enable automatic disk space management.

Type: boolean

Default: true for new clusters, false for upgraded clusters

Restart required: no

Requires that an empty file named .redpanda_data_dir be present in the data directory. If set true, Redpanda will refuse to start if the file is not found in the data directory.

Type: boolean

Default: false

Restart required: no

The maximum number of seconds that a record’s timestamp can be ahead of a Redpanda broker’s clock and still be used when deciding whether to clean up the record for data retention. This property makes possible the timely cleanup of records from clients with clocks that are drastically unsynchronized relative to Redpanda.

When determining whether to clean up a record with timestamp more than storage_ignore_timestamps_in_future_sec seconds ahead of the broker, Redpanda ignores the record’s timestamp and instead uses a valid timestamp of another record in the same segment, or (if another record’s valid timestamp is unavailable) the timestamp of when the segment file was last modified (mtime).

By default, storage_ignore_timestamps_in_future_sec is disabled (null).

Units: seconds

Default: null

Restart required: no

Supported versions: Redpanda v22.3 or later

Flag enabling a Redpanda cluster operator to use unsafe control characters within strings such as consumer group names or user names.

This flag applies only for Redpanda clusters v23.1 or earlier that have upgraded to v23.2 or later. Newly-created Redpanda clusters v23.2 or later ignore this property.

Type: boolean

Default: true

Restart required: no

Related properties: legacy_unsafe_log_warning_interval_sec

Period at which to log a warning about using unsafe strings containing control characters.

If unsafe strings are permitted by legacy_permit_unsafe_log_operations, a warning will be logged at an interval specified by this property.

Units: seconds

Default: 300

Restart required: no

Related properties: legacy_permit_unsafe_log_operations

URL of the cluster metrics reporter.

Default: s://m.rp.vectorized.io/v2

Allow automatic topic creation.

Type: boolean

Default: false

Restart required: no

Default number of partitions per topic.

Units: number of partitions per topic

Default: 1

Restart required: no

Default replication factor for new topics.

The topic property replication.factor overrides the value of default_topic_replications at the topic level.

Units: number of replicas per topic

Default: 1

Range: [1, …​], must be odd

Restart required: no

Target replication factor for internal topics.

Units: number of replicas per topic

Default: 3

Restart required: yes

Default maximum number of bytes per partition on disk before triggering deletion of the oldest messages. If null (the default value), no limit is applied.

The topic property retention.bytes overrides the value of retention_bytes at the topic level.

Units: bytes per partition

Default: null

Restart required: no

Describes how to recover from an invariant violation on the partition level.

Default: 0

Restart required: yes

Enable transactions (atomic writes).

Type: boolean

Default: true

Restart required: yes

The minimum threshold number of sessions to keep in the seq table. Not affected by compaction.

Default: 1000

Restart required: yes

Transaction manager’s synchronization timeout. Maximum time to wait for internal state machine to catch up before rejecting a request.

Units: milliseconds

Default: 10000 (10 sec)

Restart required: yes

Describes how to recover from an invariant violation at the transaction coordinator level.

Default: crash

Restart required: yes

Cleanup policy for a transaction coordinator topic.

Default: delete

Valid Values: compact, delete, compact,delete, none

Restart required: no

Delete segments older than this age. To ensure transaction state is retained as long as the longest-running transaction, make sure this is no less than transactional_id_expiration_ms.

Units: milliseconds

Default: 604800000 (1 week)

Restart required: no

Expiration time of producer IDs. Measured starting from the time of the last write until now for a given ID.

Units: milliseconds

Default: 604800000 (1 week)

Restart required: yes

Delay before scheduling the next check for timed out transactions.

Units: milliseconds

Default: 1000

Restart required: yes