Object Storage Properties

AWS or GCP access key. This access key is part of the credentials that Redpanda requires to authenticate with object storage services for Tiered Storage. This access key is used with the cloud_storage_secret_key to form the complete credentials required for authentication.

To authenticate using IAM roles, see cloud_storage_credentials_source.

Requires restart: No

Optional: No

Visibility: user

Type: string

Default: null

Optional API endpoint.

Requires restart: No

Optional: No

Visibility: user

Type: string

Default: null

TLS port override.

Requires restart: No

Optional: Yes

Visibility: user

Type: integer

Accepted values: [-32768, 32767]

Default: 443

AWS or GCS bucket that should be used to store data.

Requires restart: No

Optional: No

Visibility: user

Type: string

Default: null

Max size of object storage cache.

If both this property and cloud_storage_cache_size_percent are set, Redpanda uses the minimum of the two.

Units: bytes

Requires restart: No

Optional: Yes

Visibility: user

Type: integer

Accepted values: [0, 18446744073709551615]

Default: 0

Time interval to wait between cluster metadata uploads.

Units: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 3600000 (1 hour)

The source of credentials used to authenticate to object storage services. Required for AWS and GCS authentication with IAM roles.

To authenticate using access keys, see cloud_storage_access_key.

Valid values: config_file, aws_instance_metadata, sts, gcp_instance_metadata, azure_vm_instance_metadata, azure_aks_oidc_federation

Requires restart: Yes

Optional: Yes

Visibility: user

Default: config_file

Use legacy upload mode and do not start archiver_manager.

Requires restart: Yes

Nullable: No

Visibility: user

Type: boolean

Default: true

Disable TLS for all object storage connections.

Type: boolean

Default: false

Restart required: yes

Enable object storage. Must be set to true to use Tiered Storage or Remote Read Replicas.

Requires restart: No

Optional: Yes

Visibility: user

Type: boolean

Default: false

Maximum simultaneous object storage connections per shard, applicable to upload and download activities.

Units: number of simultaneous connections

Requires restart: No

Optional: Yes

Visibility: user

Type: integer

Accepted values: [-32768, 32767]

Default: 20

Validation performed before recovering a topic from object storage. In case of failure, the reason for the failure appears as ERROR lines in the Redpanda application log. For each topic, this reports errors for all partitions, but for each partition, only the first error is reported.

This property accepts the following parameters:

Example: Redpanda validates the topic kafka/panda-topic-recovery-NOT-OK and stops due to a fatal error on partition 0:

Each failing partition error message has the following format:

At the end of the process, Redpanda outputs a final ERROR message:

Requires restart: No

Required: No

Visibility: tunable

Type: string

Default: check_manifest_existence

Accepted values: [no_check, check_manifest_existence, check_manifest_and_segment_metadata]

Related topics: Whole Cluster Restore for Disaster Recovery

Number of metadata segments to validate, from newest to oldest, when cloud_storage_recovery_topic_validation_mode is set to check_manifest_and_segment_metadata.

Requires restart: No

Required: No

Visibility: tunable

Type: integer

Accepted values: [0, 4294967295]

Default: 10

AWS or GCP region that houses the bucket used for storage.

Requires restart: No

Optional: No

Visibility: user

Type: string

Default: null

AWS or GCP secret key.

Requires restart: No

Optional: No

Visibility: user

Type: string

Default: null

Path to certificate that should be used to validate server certificate during TLS handshake.

Requires restart: No

Optional: No

Visibility: user

Type: string

Default: null

When set to true, Redpanda automatically retrieves cluster metadata from a specified object storage bucket at the cluster’s first startup. This option is ideal for orchestrated deployments, such as Kubernetes. Ensure any previous cluster linked to the bucket is fully decommissioned to prevent conflicts between Tiered Storage subsystems.

Requires restart: Yes

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.

Requires restart: Yes

Optional: No

Visibility: user

Type: string

Default: null

Azure Data Lake Storage v2 port override. See also: cloud_storage_azure_adls_endpoint. Use when Hierarchical Namespaces are enabled on your storage account and you have set up a custom endpoint.

Requires restart: Yes

Optional: No

Visibility: user

Type: integer

Accepted values: [0, 65535]

Default: null

The name of the Azure container to use with Tiered Storage. If null, the property is disabled.

Type: string

Default: null

Restart required: yes

Supported versions: Redpanda v23.1 or later

Whether or not an Azure hierarchical namespace is enabled on the cloud_storage_azure_storage_account. If this property is not set, cloud_storage_azure_shared_key must be set, and each node tries to determine at startup if a hierarchical namespace is enabled. Setting this property to true disables the check and treats a hierarchical namespace as active. Setting to false disables the check and treats a hierarchical namespace as not active.

Requires restart: Yes

Nullable: Yes

Visibility: tunable

Type: boolean

Default: null

The managed identity ID to use for access to the Azure storage account. To use Azure managed identities, you must set cloud_storage_credentials_source to azure_vm_instance_metadata. See IAM Roles for more information on managed identities.

Type: string

Default: null

Restart required: no

Supported versions: Redpanda v24.1 or later

The shared key to be used for Azure Shared Key authentication with the Azure storage account configured by cloud_storage_azure_storage_account. If null, the property is disabled.

Type: string

Default: null

Restart required: yes

Supported versions: Redpanda v23.1 or later

The name of the Azure storage account to use with Tiered Storage. If null, the property is disabled.

Requires restart: Yes

Optional: No

Visibility: user

Type: string

Default: null

Optional object storage backend variant used to select API capabilities. If not supplied, this will be inferred from other configuration properties.

Requires restart: Yes

Optional: Yes

Visibility: user

Accepted values: [unknown, aws, google_s3_compat, azure, minio]

Default: unknown

The total number of requests the object storage background jobs can make during one background housekeeping run. This is a per-shard limit. Adjusting this limit can optimize object storage traffic and impact shard performance.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-2147483648, 2147483647]

Default: 5000

Minimum interval between Tiered Storage cache trims, measured in milliseconds. This setting dictates the cooldown period after a cache trim operation before another trim can occur. If a cache fetch operation requests a trim but the interval since the last trim has not yet passed, the trim will be postponed until this cooldown expires. Adjusting this interval helps manage the balance between cache size and retrieval performance.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 5000

Size of chunks of segments downloaded into object storage cache. Reduces space usage by only downloading the necessary chunk from a segment.

Unit: bytes

Requires restart: Yes

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [0, 18446744073709551615]

Default: 16777216

The directory where the cache archive is stored. This property is mandatory when cloud_storage_enabled is set to true.

Requires restart: Yes

Optional: No

Visibility: user

Type: string

Default: null

Maximum number of objects that may be held in the Tiered Storage cache. This applies simultaneously with cloud_storage_cache_size, and whichever limit is hit first will trigger trimming of the cache.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [0, 4294967295]

Default: 100000

Divide the object storage cache across the specified number of buckets. This only works for objects with randomized prefixes. The names are not changed when the value is set to zero.

Requires restart: No

Nullable: No

Visibility: tunable

Type: integer

Accepted values: [0, 4294967295]

Default: 0

Maximum size of the cloud cache as a percentage of unreserved disk space (see disk_reservation_percent). The default value for this option is tuned for a shared disk configuration. Consider increasing the value if using a dedicated cache disk.

The property cloud_storage_cache_size controls the same limit expressed as a fixed number of bytes. If both cloud_storage_cache_size and cloud_storage_cache_size_percent are set, Redpanda uses the minimum of the two.

Units: percentage of total disk size.

Requires restart: No

Optional: No

Visibility: user

Type: number

Default: 20.0

The cache performs a recursive directory inspection during the cache trim. The information obtained during the inspection can be carried over to the next trim operation. This property sets a limit on the memory occupied by objects that can be carried over from one trim to next, and it allows the cache to quickly unblock readers before starting the directory inspection.

Requires restart: No

Nullable: No

Visibility: tunable

Type: integer

Accepted values: [0, 4294967295]

Default: 262144

The maximum number of concurrent tasks launched for traversing the directory structure during cache trimming. A higher number allows cache trimming to run faster but can cause latency spikes due to increased pressure on I/O subsystem and syscall threads.

Requires restart: No

Nullable: No

Visibility: tunable

Type: integer

Accepted values: [0, 65535]

Default: 1

Selects a strategy for evicting unused cache chunks.

Requires restart: No

Optional: Yes

Visibility: tunable

Accepted values: [eager, capped, predictive]

Default: eager

Number of chunks to prefetch ahead of every downloaded chunk. Prefetching additional chunks can enhance read performance by reducing wait times for sequential data access. A value of 0 disables prefetching, relying solely on on-demand downloads. Adjusting this property allows for tuning the balance between improved read performance and increased network and storage I/O.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [0, 65535]

Default: 0

Number of groups to upload in a single snapshot object during consumer offsets upload. Setting a lower value means a larger number of smaller snapshots are uploaded.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Default: 1000

Number of attempts metadata operations may be retried.

Requires restart: Yes

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-32768, 32767]

Default: 5

Timeout for cluster metadata uploads.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 60000

The hostname to connect to for retrieving role based credentials. Derived from cloud_storage_credentials_source if not set. Only required when using IAM role-based access.

Requires restart: Yes

Optional: No

Visibility: tunable

Type:

Accepted values: [config_file, aws_instance_metadata, sts, gcp_instance_metadata, azure_aks_oidc_federation, azure_vm_instance_metadata]

Default: config_file

Disable chunk reads and switch back to legacy mode where full segments are downloaded. When set to true, this option disables the more efficient chunk-based reads, causing Redpanda to download entire segments. This legacy behavior might be useful in specific scenarios where chunk-based fetching is not optimal.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Disable all metadata consistency checks to allow Redpanda to replay logs with inconsistent Tiered Storage metadata. This option should generally remain disabled, except for new clusters.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: true

Begins the read replica sync loop in topic partitions with Tiered Storage enabled. The property exists to simplify testing and shouldn’t be set in production.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Disable all upload consistency checks to allow Redpanda to upload logs with gaps and replicate metadata with consistency violations. Normally, this option should be disabled.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Begins the upload loop in topic partitions with Tiered Storage enabled. The property exists to simplify testing and shouldn’t be set in production.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Enable re-uploading data for compacted topics. When set to true, Redpanda can re-upload data for compacted topics to object storage, ensuring that the most current state of compacted topics is available in the cloud. Disabling this property (false) may reduce storage and network overhead but at the risk of not having the latest compacted data state in object storage.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: true

Default remote read config value for new topics. When set to true, new topics are by default configured to allow reading data directly from object storage, facilitating access to older data that might have been offloaded as part of Tiered Storage. With the default set to false, remote reads must be explicitly enabled at the topic level.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Default remote write value for new topics. When set to true, new topics are by default configured to upload data to object storage. With the default set to false, remote write must be explicitly enabled at the topic level.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Enable routine checks (scrubbing) of object storage partitions. The scrubber validates the integrity of data and metadata uploaded to object storage.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: false

Enables adjacent segment merging. The segments are reuploaded if there is an opportunity for that and if it will improve the performance of Tiered Storage.

Related topics:

Requires restart: No

Optional: Yes

Visibility: tunable

Type: boolean

Default: true

Interval, in milliseconds, between a final scrub and the next scrub.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 4320000000 (12h)

Timeout for running the cloud storage garbage collection, in milliseconds.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 30000

Time limit on waiting for uploads to complete before a leadership transfer. If this is null, leadership transfers proceed without waiting.

Unit: milliseconds

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 5000

Interval, in milliseconds, between object storage housekeeping tasks.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 5000

The maximum number of chunks per segment that can be hydrated at a time. Above this number, unused chunks are trimmed.

A segment is divided into chunks. Chunk hydration means downloading the chunk (which is a small part of a full segment) from cloud storage and placing it in the local disk cache. Redpanda periodically removes old, unused chunks from your local disk. This process is called chunk eviction. This property controls how many chunks can be present for a given segment in local disk at a time, before eviction is triggered, removing the oldest ones from disk. Note that this property is not used for the default eviction strategy which simply removes all unused chunks.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: number

Accepted values: [0, 1]

Default: 0.7

Time to wait for a hydration request to be fulfilled. If hydration is not completed within this time, the consumer is notified with a timeout error.

Negative doesn’t make sense, but it may not be checked-for/enforced. Large is subjective, but a huge timeout also doesn’t make sense. This particular config doesn’t have a min/max bounds control, but it probably should to avoid mistakes.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [0, 17592186044415]

Default: 600000

The object storage request rate threshold for idle state detection. If the average request rate for the configured period is lower than this threshold, the object storage is considered idle.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: number

Default: 10.0

The timeout, in milliseconds, used to detect the idle state of the object storage API. If the average object storage request rate is below this threshold for a configured amount of time, the object storage is considered idle and the housekeeping jobs are started.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 10000

Initial backoff time for exponential backoff algorithm (ms).

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 100

Amount of memory that can be used to handle Tiered Storage metadata.

Unit: bytes

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Default: 1048576

The interval, in milliseconds, determines how long the materialized manifest can stay in the cache under contention. This setting is used for performance tuning. When the spillover manifest is materialized and stored in the cache, and the cache needs to evict it, it uses this value as a timeout. The cursor that uses the spillover manifest uses this value as a TTL interval, after which it stops referencing the manifest making it available for eviction. This only affects spillover manifests under contention.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 10000

Minimum interval, in seconds, between partition manifest uploads. Actual time between uploads may be greater than this interval. If this is null, metadata is updated after each segment upload.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Accepted values: [-17179869184, 17179869183]

Default: 60

Manifest upload timeout, in milliseconds.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 10000

Maximum concurrent segment hydrations of remote data per CPU core. If unset, value of cloud_storage_max_connections / 2 is used, which means that half of available object storage bandwidth could be used to download data from object storage. If the cloud storage cache is empty every new segment reader will require a download. This will lead to 1:1 mapping between number of partitions scanned by the fetch request and number of parallel downloads. If this value is too large the downloads can affect other workloads. In case of any problem caused by the tiered-storage reads this value can be lowered. This will only affect segment hydrations (downloads) but won’t affect cached segments. If fetch request is reading from the tiered-storage cache its concurrency will only be limited by available memory.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Accepted values: [0, 4294967295]

Default: null

Defines the maximum duration an HTTPS connection to object storage can stay idle, in milliseconds, before being terminated. This setting reduces resource utilization by closing inactive connections. Adjust this property to balance keeping connections ready for subsequent requests and freeing resources associated with idle connections.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 5000

Maximum concurrent I/O cursors of materialized remote segments per CPU core. If unset, the value of topic_partitions_per_shard is used, where one segment reader per partition is used if the shard is at its maximum partition capacity. These readers are cached across Kafka consume requests and store a readahead buffer.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Accepted values: [0, 4294967295]

Default: null

The per-partition limit for the number of segments pending deletion from the cloud. Segments can be deleted due to retention or compaction. If this limit is breached and deletion fails, then segments are orphaned in the cloud and must be removed manually.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Default: 5000

Maximum bandwidth allocated to Tiered Storage operations per shard, in bytes per second. This setting limits the Tiered Storage subsystem’s throughput per shard, facilitating precise control over bandwidth usage in testing scenarios. In production environments, use cloud_storage_throughput_limit_percent for more dynamic throughput management based on actual storage capabilities.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Default: 1073741824

Timeout for Tiered Storage metadata synchronization.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 10000

The minimum number of chunks per segment for trimming to be enabled. If the number of chunks in a segment is below this threshold, the segment is small enough that all chunks in it can be hydrated at any given time.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [0, 18446744073709551615]

Default: 5

Time interval between two partial scrubs of the same partition.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 3600000 (1h)

Timeout to check if new data is available for partitions in object storage for read replicas.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 30000

Retention in bytes for topics created during automated recovery.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Default: 1073741824

Timeout for IAM role related operations, in milliseconds.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 30000

Jitter applied to the object storage scrubbing interval.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 600000 (10min)

Time that a segment can be kept locally without uploading it to the object storage, in seconds.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Accepted values: [-17179869184, 17179869183]

Default: 3600 (one hour)

Smallest acceptable segment size in the object storage. Default: cloud_storage_segment_size_target/2.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Default: cloud_storage_segment_size_target/2

Related property: cloud_storage_segment_size_target

Desired segment size in the object storage. The default is set in the topic-level segment.bytes property.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Default: null

Log segment upload timeout, in milliseconds.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 30000

Maximum number of segments in the spillover manifest that can be offloaded to the object storage. This setting serves as a threshold for triggering data offload based on the number of segments, rather than the total size of the manifest. It is designed for use in testing environments to control the offload behavior more granularly. In production settings, manage offloads based on the manifest size through cloud_storage_spillover_manifest_size for more predictable outcomes.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Default: null

The size of the manifest which can be offloaded to the cloud. If the size of the local manifest stored in Redpanda exceeds cloud_storage_spillover_manifest_size by two times the spillover mechanism will split the manifest into two parts and one will be uploaded to object storage.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Default: 65536

Maximum throughput used by Tiered Storage per broker expressed as a percentage of the disk bandwidth. If the server has several disks, Redpanda uses the one that stores the Tiered Storage cache. Even if Tiered Storage is allowed to use the full bandwidth of the disk (100%), it won’t necessarily use it in full. The actual usage depends on your workload and the state of the Tiered Storage cache. This setting is a safeguard that prevents Tiered Storage from using too many system resources: it is not a performance tuning knob.

Requires restart: No

Optional: No

Visibility: tunable

Type: integer

Default: 50

Grace period during which the purger refuses to purge the topic.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 30000

Derivative coefficient for upload PID controller.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: number

Default: 0.0

Maximum number of I/O and CPU shares that archival upload can use.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-32768, 32767]

Default: 1000

Minimum number of I/O and CPU shares that archival upload can use.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-32768, 32767]

Default: 100

Proportional coefficient for upload PID controller.

Requires restart: No

Optional: Yes

Visibility: tunable

Type: number

Default: -2.0

Initial backoff interval when there is nothing to upload for a partition, in milliseconds.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 100

Maximum backoff interval when there is nothing to upload for a partition, in milliseconds.

Unit: milliseconds

Requires restart: No

Optional: Yes

Visibility: tunable

Type: integer

Accepted values: [-17592186044416, 17592186044415]

Default: 10000