Object Storage Properties
Object storage properties are a type of cluster property. For information on how to edit cluster properties, see Configure Cluster Properties.
Some object storage properties require that you restart the cluster for any updates to take effect. See the specific property details to identify whether or not a restart is required. |
Cloud configuration
Object storage properties should only be set if you enable Tiered Storage.
cloud_storage_access_key
AWS or GCP access key. This access key is part of the credentials that Redpanda requires to authenticate with object storage services for Tiered Storage. This access key is used with the cloud_storage_secret_key
to form the complete credentials required for authentication.
To authenticate using IAM roles, see cloud_storage_credentials_source
.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_api_endpoint
Optional API endpoint. The only instance in which you must set this value is when using a custom domain with your object storage service.
-
AWS: If not set, this is automatically generated using region and bucket. Otherwise, this uses the value assigned.
-
GCP: If not set, this is automatically generated using
storage.googleapis.com
and bucket. -
Azure: If not set, this is automatically generated using
blob.core.windows.net
andcloud_storage_azure_storage_account
. If you have enabled hierarchical namespaces for your storage account and use a custom endpoint, usecloud_storage_azure_adls_endpoint
.
Requires restart: No
Optional: Yes (if not using a custom domain)
Visibility: user
Type: string
Default: null
cloud_storage_api_endpoint_port
TLS port override.
Requires restart: Yes
Visibility: user
Type: integer
Accepted values: [-32768
, 32767
]
Default: 443
cloud_storage_attempt_cluster_restore_on_bootstrap
When set to true
, Redpanda automatically retrieves cluster metadata from a specified object storage bucket at the cluster’s first startup. This option is ideal for orchestrated deployments, such as Kubernetes. Ensure any previous cluster linked to the bucket is fully decommissioned to prevent conflicts between Tiered Storage subsystems.
Requires restart: Yes
Visibility: tunable
Type: boolean
Default: false
cloud_storage_azure_adls_endpoint
Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.
If not set, this is automatically generated using dfs.core.windows.net
and cloud_storage_azure_storage_account
.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_azure_adls_port
Azure Data Lake Storage v2 port override. See also: cloud_storage_azure_adls_endpoint
. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.
Requires restart: Yes
Visibility: user
Type: integer
Accepted values: [0
, 65535
]
Default: null
cloud_storage_azure_container
The name of the Azure container to use with Tiered Storage. If null
, the property is disabled.
The container must belong to cloud_storage_azure_storage_account .
|
Type: string
Default: null
Restart required: yes
Supported versions: Redpanda v23.1 or later
cloud_storage_azure_hierarchical_namespace_enabled
Force Redpanda to use or not use an Azure Data Lake Storage (ADLS) Gen2 hierarchical namespace-compliant client in cloud_storage_azure_storage_account
.
When this property is not set, cloud_storage_azure_shared_key
must be set, and each broker checks at startup if a hierarchical namespace is enabled.
When set to true
, this property disables the check and assumes a hierarchical namespace is enabled.
When set to false
, this property disables the check and assumes a hierarchical namespace is not enabled.
This setting should be used only in emergencies where Redpanda fails to detect the correct a hierarchical namespace status.
Requires restart: Yes
Visibility: tunable
Type: boolean
Default: null
cloud_storage_azure_managed_identity_id
The managed identity ID to use for access to the Azure storage account. To use Azure managed identities, you must set cloud_storage_credentials_source
to azure_vm_instance_metadata
. See IAM Roles for more information on managed identities.
Type: string
Default: null
Restart required: no
Supported versions: Redpanda v24.1 or later
cloud_storage_azure_shared_key
The account access key to be used for Azure Shared Key authentication with the Azure storage account configured by cloud_storage_azure_storage_account
. If null
, the property is disabled.
Redpanda expects this key string to be Base64 encoded. |
Type: string
Default: null
Restart required: yes
Supported versions: Redpanda v23.1 or later
cloud_storage_azure_storage_account
The name of the Azure storage account to use with Tiered Storage. If null
, the property is disabled.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_backend
Optional object storage backend variant used to select API capabilities. If not supplied, this will be inferred from other configuration properties.
Requires restart: Yes
Visibility: user
Accepted values: [unknown
, aws
, google_s3_compat
, azure
, minio
]
Default: unknown
cloud_storage_background_jobs_quota
The total number of requests the object storage background jobs can make during one background housekeeping run. This is a per-shard limit. Adjusting this limit can optimize object storage traffic and impact shard performance.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-2147483648
, 2147483647
]
Default: 5000
cloud_storage_bucket
AWS or GCP bucket that should be used to store data.
Modifying this property after writing data to a bucket could cause data loss. |
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_cache_check_interval
Minimum interval between Tiered Storage cache trims, measured in milliseconds. This setting dictates the cooldown period after a cache trim operation before another trim can occur. If a cache fetch operation requests a trim but the interval since the last trim has not yet passed, the trim will be postponed until this cooldown expires. Adjusting this interval helps manage the balance between cache size and retrieval performance.
Unit: milliseconds
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 5000
cloud_storage_cache_chunk_size
Size of chunks of segments downloaded into object storage cache. Reduces space usage by only downloading the necessary chunk from a segment.
Unit: bytes
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [0
, 18446744073709551615
]
Default: 16777216
cloud_storage_cache_directory
The directory where the cache archive is stored. This property is mandatory when cloud_storage_enabled
is set to true
.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_cache_max_objects
Maximum number of objects that may be held in the Tiered Storage cache. This applies simultaneously with cloud_storage_cache_size
, and whichever limit is hit first will trigger trimming of the cache.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 4294967295
]
Default: 100000
cloud_storage_cache_num_buckets
Divide the object storage cache across the specified number of buckets. This only works for objects with randomized prefixes. The names are not changed when the value is set to zero.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 4294967295
]
Default: 0
cloud_storage_cache_size
Maximum size of the object storage cache, in bytes.
This property works together with cloud_storage_cache_size_percent
to define cache behavior:
-
When both properties are set, Redpanda uses the smaller calculated value of the two, in bytes.
-
If one of these properties is set to
0
, Redpanda uses the non-zero value. -
These properties cannot both be
0
. -
cloud_storage_cache_size
cannot be0
whilecloud_storage_cache_size_percent
isnull
.
Unit: bytes
Requires restart: No
Visibility: user
Type: integer
Accepted values: [0
, 18446744073709551615
]
Default: 0
cloud_storage_cache_size_percent
Maximum size of the cache as a percentage, minus the space that Redpanda avoids using defined by the disk_reservation_percent
cluster property. This is calculated at startup and dynamically updated if either this property, disk_reservation_percent
, or cloud_storage_cache_size
changes.
This property works together with cloud_storage_cache_size
to define cache behavior:
-
When both properties are set, Redpanda uses the smaller calculated value of the two, in bytes.
-
If one of these properties is set to
0
, Redpanda uses the non-zero value. -
These properties cannot both be
0
. -
cloud_storage_cache_size
cannot be0
whilecloud_storage_cache_size_percent
isnull
.
Unit: percent
Requires restart: No
Visibility: user
Type: number
Default: 20.0
cloud_storage_cache_trim_threshold_percent_objects
Introduced in 24.1.10.
Cache trimming is triggered when the number of objects in the cache reaches this percentage relative to its maximum object count. If unset, the default behavior is to start trimming when the cache is full.
Requires restart: No
Visibility: tunable
Type: number
Default: null
cloud_storage_cache_trim_threshold_percent_size
Introduced in 24.1.10.
Cache trimming is triggered when the cache size reaches this percentage relative to its maximum capacity. If unset, the default behavior is to start trimming when the cache is full.
Requires restart: No
Visibility: tunable
Type: number
Default: null
cloud_storage_cache_trim_walk_concurrency
The maximum number of concurrent tasks launched for traversing the directory structure during cache trimming. A higher number allows cache trimming to run faster but can cause latency spikes due to increased pressure on I/O subsystem and syscall threads.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 65535
]
Default: 1
cloud_storage_chunk_eviction_strategy
Selects a strategy for evicting unused cache chunks.
Requires restart: No
Visibility: tunable
Accepted values: [eager
, capped
, predictive
]
Default: eager
cloud_storage_chunk_prefetch
Number of chunks to prefetch ahead of every downloaded chunk. Prefetching additional chunks can enhance read performance by reducing wait times for sequential data access. A value of 0
disables prefetching, relying solely on on-demand downloads. Adjusting this property allows for tuning the balance between improved read performance and increased network and storage I/O.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 65535
]
Default: 0
cloud_storage_cluster_metadata_num_consumer_groups_per_upload
Number of groups to upload in a single snapshot object during consumer offsets upload. Setting a lower value will mean a larger number of smaller snapshots are uploaded.
Requires restart: No
Visibility: tunable
Type: integer
Default: 1000
cloud_storage_cluster_metadata_retries
Number of attempts metadata operations may be retried.
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-32768
, 32767
]
Default: 5
cloud_storage_cluster_metadata_upload_interval_ms
Time interval to wait between cluster metadata uploads.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 3600000
cloud_storage_cluster_metadata_upload_timeout_ms
Timeout for cluster metadata uploads.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 60000
cloud_storage_credentials_host
The hostname to connect to for retrieving role based credentials. Derived from cloud_storage_credentials_source
if not set. Only required when using IAM role based access. To authenticate using access keys, see cloud_storage_access_key
.
Requires restart: Yes
Visibility: tunable
Type:
Accepted values: [config_file
, aws_instance_metadata
, sts
, gcp_instance_metadata
, azure_aks_oidc_federation
, azure_vm_instance_metadata
]
Default: config_file
cloud_storage_credentials_source
The source of credentials used to authenticate to object storage services. Required for AWS or GCP authentication with IAM roles.
To authenticate using access keys, see cloud_storage_access_key
.
Accepted values: config_file
, aws_instance_metadata
, sts, gcp_instance_metadata
, azure_vm_instance_metadata
, azure_aks_oidc_federation
Requires restart: Yes
Visibility: user
Default: config_file
cloud_storage_crl_file
Path to certificate revocation list for cloud_storage_trust_file
.
Requires restart: No
Visibility: user
Type: string
Default: null
cloud_storage_disable_chunk_reads
Disable chunk reads and switch back to legacy mode where full segments are downloaded. When set to true
, this option disables the more efficient chunk-based reads, causing Redpanda to download entire segments. This legacy behavior might be useful in specific scenarios where chunk-based fetching is not optimal.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_disable_read_replica_loop_for_tests
Begins the read replica sync loop in topic partitions with Tiered Storage enabled. The property exists to simplify testing and shouldn’t be set in production.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_disable_remote_labels_for_tests
If true
, Redpanda disables remote labels and falls back on the hash-based object naming scheme for new topics.
This property exists to simplify testing and shouldn’t be set in production. |
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_disable_tls
Disable TLS for all object storage connections.
Requires restart: Yes
Visibility: user
Type: boolean
Default: false
cloud_storage_disable_upload_consistency_checks
Disable all upload consistency checks to allow Redpanda to upload logs with gaps and replicate metadata with consistency violations. Do not change the default value unless requested by Redpanda Support.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_disable_upload_loop_for_tests
Begins the upload loop in topic partitions with Tiered Storage enabled. The property exists to simplify testing and shouldn’t be set in production.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_enable_compacted_topic_reupload
Enable re-uploading data for compacted topics.
When set to true
, Redpanda can re-upload data for compacted topics to object storage, ensuring that the most current state of compacted topics is available in the cloud. Disabling this property (false
) may reduce storage and network overhead but at the risk of not having the latest compacted data state in object storage.
Requires restart: No
Visibility: tunable
Type: boolean
Default: true
cloud_storage_enable_remote_allow_gaps
Controls the eviction of locally stored log segments when Tiered Storage uploads are paused. Set to false
to only evict data that has already been uploaded to object storage. If the retained data fills the local volume, Redpanda throttles producers. Set to true
to allow the eviction of locally stored log segments, which may create gaps in offsets.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_enable_remote_read
Default remote read config value for new topics.
When set to true
, new topics are by default configured to allow reading data directly from object storage, facilitating access to older data that might have been offloaded as part of Tiered Storage. With the default set to false
, remote reads must be explicitly enabled at the topic level.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_enable_remote_write
Default remote write value for new topics.
When set to true
, new topics are by default configured to upload data to object storage. With the default set to false
, remote write must be explicitly enabled at the topic level.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_enable_scrubbing
Enable routine checks (scrubbing) of object storage partitions. The scrubber validates the integrity of data and metadata uploaded to object storage.
Requires restart: No
Visibility: tunable
Type: boolean
Default: false
cloud_storage_enable_segment_merging
Enables adjacent segment merging. The segments are reuploaded if there is an opportunity for that and if it will improve the performance of Tiered Storage.
Requires restart: No
Visibility: tunable
Type: boolean
Default: true
Related topics:
cloud_storage_enable_segment_uploads
Controls the upload of log segments to Tiered Storage. If set to false
, this property temporarily pauses all log segment uploads from the Redpanda cluster. When the uploads are paused, the cloud_storage_enable_remote_allow_gaps
cluster configuration and redpanda.remote.allowgaps
topic properties control local retention behavior.
Requires restart: No
Visibility: tunable
Type: boolean
Default: true
cloud_storage_enabled
Some values for this property require an Enterprise license. Refer to the Enterprise license required field for specific requirements. For license details, see Redpanda Licenses and Enterprise Features. |
Enable object storage. Must be set to true
to use Tiered Storage or Remote Read Replicas.
Requires restart: Yes
Visibility: user
Type: boolean
Enterprise license required: true
Default: false
cloud_storage_full_scrub_interval_ms
Interval, in milliseconds, between a final scrub and the next scrub.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 43200000
(12h)
cloud_storage_garbage_collect_timeout_ms
Timeout for running the cloud storage garbage collection, in milliseconds.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 30000
cloud_storage_graceful_transfer_timeout_ms
Time limit on waiting for uploads to complete before a leadership transfer. If this is null
, leadership transfers proceed without waiting.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 5000
cloud_storage_housekeeping_interval_ms
Interval, in milliseconds, between object storage housekeeping tasks.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 300000
cloud_storage_hydrated_chunks_per_segment_ratio
The maximum number of chunks per segment that can be hydrated at a time. Above this number, unused chunks are trimmed.
A segment is divided into chunks. Chunk hydration means downloading the chunk (which is a small part of a full segment) from cloud storage and placing it in the local disk cache. Redpanda periodically removes old, unused chunks from your local disk. This process is called chunk eviction. This property controls how many chunks can be present for a given segment in local disk at a time, before eviction is triggered, removing the oldest ones from disk. Note that this property is not used for the default eviction strategy which simply removes all unused chunks.
Requires restart: No
Visibility: tunable
Type: number
Accepted values: [0
, 1
]
Default: 0.7
cloud_storage_hydration_timeout_ms
Time to wait for a hydration request to be fulfilled. If hydration is not completed within this time, the consumer is notified with a timeout error.
Negative doesn’t make sense, but it may not be checked-for/enforced. Large is subjective, but a huge timeout also doesn’t make sense. This particular config doesn’t have a min/max bounds control, but it probably should to avoid mistakes.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 17592186044415
]
Default: 600000
cloud_storage_idle_threshold_rps
The object storage request rate threshold for idle state detection. If the average request rate for the configured period is lower than this threshold, the object storage is considered idle.
Requires restart: No
Visibility: tunable
Type: number
Default: 10.0
cloud_storage_idle_timeout_ms
The timeout, in milliseconds, used to detect the idle state of the object storage API. If the average object storage request rate is below this threshold for a configured amount of time, the object storage is considered idle and the housekeeping jobs are started.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 10000
cloud_storage_initial_backoff_ms
Initial backoff time for exponential backoff algorithm (ms).
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 100
cloud_storage_inventory_based_scrub_enabled
Scrubber uses the latest cloud storage inventory report, if available, to check if the required objects exist in the bucket or container.
Requires restart: Yes
Visibility: tunable
Type: boolean
Default: false
cloud_storage_inventory_hash_path_directory
Directory to store inventory report hashes for use by cloud storage scrubber.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_inventory_id
The name of the scheduled inventory job created by Redpanda to generate bucket or container inventory reports.
Requires restart: Yes
Visibility: tunable
Type: string
Default: redpanda_scrubber_inventory
cloud_storage_inventory_max_hash_size_during_parse
Maximum bytes of hashes held in memory before writing data to disk during inventory report parsing. This affects the number of files written to disk during inventory report parsing. When this limit is reached, new files are written to disk.
Unit: bytes
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 18446744073709551615
]
Default: 67108864
cloud_storage_inventory_report_check_interval_ms
Time interval between checks for a new inventory report in the cloud storage bucket or container.
Unit: milliseconds
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 21600000
(6h)
cloud_storage_inventory_reports_prefix
The prefix to the path in the cloud storage bucket or container where inventory reports will be placed.
Requires restart: Yes
Visibility: tunable
Type: string
Default: redpanda_scrubber_inventory
cloud_storage_inventory_self_managed_report_config
If enabled, Redpanda will not attempt to create the scheduled report configuration using cloud storage APIs. The scrubbing process will look for reports in the expected paths in the bucket or container, and use the latest report found. Primarily intended for use in testing and on backends where scheduled inventory reports are not supported.
Requires restart: Yes
Visibility: tunable
Type: boolean
Default: false
cloud_storage_manifest_cache_size
Amount of memory that can be used to handle Tiered Storage metadata.
Unit: bytes
Requires restart: No
Visibility: tunable
Type: integer
Default: 1048576
cloud_storage_materialized_manifest_ttl_ms
The interval, in milliseconds, determines how long the materialized manifest can stay in the cache under contention. This setting is used for performance tuning. When the spillover manifest is materialized and stored in the cache, and the cache needs to evict it, it uses this value as a timeout. The cursor that uses the spillover manifest uses this value as a TTL interval, after which it stops referencing the manifest making it available for eviction. This only affects spillover manifests under contention.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 10000
cloud_storage_manifest_max_upload_interval_sec
Minimum interval, in seconds, between partition manifest uploads. Actual time between uploads may be greater than this interval. If this is null
, metadata is updated after each segment upload.
Unit: seconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17179869184
, 17179869183
]
Default: 60
cloud_storage_manifest_upload_timeout_ms
Manifest upload timeout, in milliseconds.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 10000
cloud_storage_max_concurrent_hydrations_per_shard
Maximum concurrent segment hydrations of remote data per CPU core. If unset, value of cloud_storage_max_connections / 2
is used, which means that half of available object storage bandwidth could be used to download data from object storage. If the cloud storage cache is empty every new segment reader will require a download. This will lead to 1:1 mapping between number of partitions scanned by the fetch request and number of parallel downloads. If this value is too large the downloads can affect other workloads. In case of any problem caused by the tiered-storage reads this value can be lowered. This will only affect segment hydrations (downloads) but won’t affect cached segments. If fetch request is reading from the tiered-storage cache its concurrency will only be limited by available memory.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 4294967295
]
Default: null
cloud_storage_max_connection_idle_time_ms
Defines the maximum duration an HTTPS connection to object storage can stay idle, in milliseconds, before being terminated. This setting reduces resource utilization by closing inactive connections. Adjust this property to balance keeping connections ready for subsequent requests and freeing resources associated with idle connections.
Unit: milliseconds
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 5000
cloud_storage_max_connections
Maximum simultaneous object storage connections per shard, applicable to upload and download activities.
Requires restart: Yes
Visibility: user
Type: integer
Accepted values: [-32768
, 32767
]
Default: 20
cloud_storage_max_segment_readers_per_shard
Maximum concurrent I/O cursors of materialized remote segments per CPU core. If unset, the value of topic_partitions_per_shard
is used, where one segment reader per partition is used if the shard is at its maximum partition capacity. These readers are cached across Kafka consume requests and store a readahead buffer.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 4294967295
]
Default: null
cloud_storage_max_segments_pending_deletion_per_partition
The per-partition limit for the number of segments pending deletion from the cloud. Segments can be deleted due to retention or compaction. If this limit is breached and deletion fails, then segments are orphaned in the cloud and must be removed manually.
Requires restart: No
Visibility: tunable
Type: integer
Default: 5000
cloud_storage_max_throughput_per_shard
Maximum bandwidth allocated to Tiered Storage operations per shard, in bytes per second.
This setting limits the Tiered Storage subsystem’s throughput per shard, facilitating precise control over bandwidth usage in testing scenarios. In production environments, use cloud_storage_throughput_limit_percent
for more dynamic throughput management based on actual storage capabilities.
Requires restart: No
Visibility: tunable
Type: integer
Default: 1073741824
cloud_storage_metadata_sync_timeout_ms
Timeout for Use Tiered Storage metadata synchronization.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 10000
cloud_storage_min_chunks_per_segment_threshold
The minimum number of chunks per segment for trimming to be enabled. If the number of chunks in a segment is below this threshold, the segment is small enough that all chunks in it can be hydrated at any given time.
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [0
, 18446744073709551615
]
Default: 5
cloud_storage_partial_scrub_interval_ms
Time interval between two partial scrubs of the same partition.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 3600000
(1h)
cloud_storage_readreplica_manifest_sync_timeout_ms
Timeout to check if new data is available for partitions in object storage for read replicas.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 30000
cloud_storage_recovery_temporary_retention_bytes_default
Retention in bytes for topics created during automated recovery.
Requires restart: No
Visibility: tunable
Type: integer
Default: 1073741824
cloud_storage_recovery_topic_validation_depth
Number of metadata segments to validate, from newest to oldest, when cloud_storage_recovery_topic_validation_mode
is set to check_manifest_and_segment_metadata
.
Requires restart: No
Required: No
Visibility: tunable
Type: integer
Accepted values: [0
, 4294967295
]
Default: 10
cloud_storage_recovery_topic_validation_mode
Validation performed before recovering a topic from object storage. In case of failure, the reason for the failure appears as ERROR
lines in the Redpanda application log. For each topic, this reports errors for all partitions, but for each partition, only the first error is reported.
This property accepts the following parameters:
-
no_check
: Skips the checks for topic recovery. -
check_manifest_existence
: Runs an existence check on eachpartition_manifest
. Fails if there are connection issues to the object storage. -
check_manifest_and_segment_metadata
: Downloads the manifest and runs a consistency check, comparing the metadata with the cloud storage objects. The process fails if metadata references any missing cloud storage objects.
Example: Redpanda validates the topic kafka/panda-topic-recovery-NOT-OK
and stops due to a fatal error on partition 0:
ERROR 2024-04-24 21:29:08,166 [shard 1:main] cluster - [fiber11|0|299996ms recovery validation of {kafka/panda-topic-recovery-NOT-OK/0}/24] - manifest metadata check: missing segment, validation not ok
ERROR 2024-04-24 21:29:08,166 [shard 1:main] cluster - topics_frontend.cc:519 - Stopping recovery of {kafka/panda-topic-recovery-NOT-OK} due to validation error
Each failing partition error message has the following format:
ERROR .... [... recovery validation of {<namespace/topic/partition>}...] - <failure-reason>, validation not ok
At the end of the process, Redpanda outputs a final ERROR message:
ERROR ... ... - Stopping recovery of {<namespace/topic>} due to validation error
Requires restart: No
Required: No
Visibility: tunable
Type: string
Default: check_manifest_existence
Accepted values: [no_check
, check_manifest_existence
, check_manifest_and_segment_metadata
]
Related topics: Whole Cluster Restore for Disaster Recovery
cloud_storage_region
Cloud provider region that houses the bucket or container used for storage.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_roles_operation_timeout_ms
Timeout for IAM role related operations (ms).
Unit: milliseconds
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 30000
cloud_storage_scrubbing_interval_jitter_ms
Jitter applied to the object storage scrubbing interval.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 600000
cloud_storage_secret_key
Cloud provider secret key.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_segment_max_upload_interval_sec
Time that a segment can be kept locally without uploading it to the object storage, in seconds.
Unit: seconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17179869184
, 17179869183
]
Default: 3600
(one hour)
cloud_storage_segment_size_min
Smallest acceptable segment size in the object storage. Default: cloud_storage_segment_size_target
/2.
Requires restart: No
Visibility: tunable
Type: integer
Default: cloud_storage_segment_size_target/2
Related property: cloud_storage_segment_size_target
cloud_storage_segment_size_target
Desired segment size in the object storage. The default is set in the topic-level segment.bytes
property.
Requires restart: No
Visibility: tunable
Type: integer
Default: null
cloud_storage_segment_upload_timeout_ms
Log segment upload timeout, in milliseconds.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 30000
cloud_storage_spillover_manifest_max_segments
Maximum number of segments in the spillover manifest that can be offloaded to the object storage. This setting serves as a threshold for triggering data offload based on the number of segments, rather than the total size of the manifest. It is designed for use in testing environments to control the offload behavior more granularly. In production settings, manage offloads based on the manifest size through cloud_storage_spillover_manifest_size
for more predictable outcomes.
Requires restart: No
Visibility: tunable
Type: integer
Default: null
cloud_storage_spillover_manifest_size
The size of the manifest which can be offloaded to the cloud. If the size of the local manifest stored in Redpanda exceeds cloud_storage_spillover_manifest_size
by two times the spillover mechanism will split the manifest into two parts and one will be uploaded to object storage.
Requires restart: No
Visibility: tunable
Type: integer
Default: 65536
cloud_storage_throughput_limit_percent
Maximum throughput used by Tiered Storage per broker expressed as a percentage of the disk bandwidth. If the server has several disks, Redpanda uses the one that stores the Tiered Storage cache. Even if Tiered Storage is allowed to use the full bandwidth of the disk (100%), it won’t necessarily use it in full. The actual usage depends on your workload and the state of the Tiered Storage cache. This setting is a safeguard that prevents Tiered Storage from using too many system resources: it is not a performance tuning knob.
Unit: percent
Requires restart: No
Visibility: tunable
Type: integer
Default: 50
cloud_storage_topic_purge_grace_period_ms
Grace period during which the purger refuses to purge the topic.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 30000
cloud_storage_trust_file
Path to certificate that should be used to validate server certificate during TLS handshake.
Requires restart: Yes
Visibility: user
Type: string
Default: null
cloud_storage_upload_ctrl_d_coeff
Derivative coefficient for upload PID controller.
Requires restart: Yes
Visibility: tunable
Type: number
Default: 0.0
cloud_storage_upload_ctrl_max_shares
Maximum number of I/O and CPU shares that archival upload can use.
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-32768
, 32767
]
Default: 1000
cloud_storage_upload_ctrl_min_shares
Minimum number of I/O and CPU shares that archival upload can use.
Requires restart: Yes
Visibility: tunable
Type: integer
Accepted values: [-32768
, 32767
]
Default: 100
cloud_storage_upload_ctrl_p_coeff
Proportional coefficient for upload PID controller.
Requires restart: Yes
Visibility: tunable
Type: number
Default: -2.0
cloud_storage_upload_ctrl_update_interval_ms
The interval (in milliseconds) for updating the controller that manages the priority of Tiered Storage uploads. This property determines how frequently the system recalculates and adjusts the work scheduling for uploads to object storage.
This is an internal-only configuration and should be enabled only after consulting with Redpanda support.
Requires restart: Yes
Visibility: tunable
Type: number
Default: 60000
(60s)
cloud_storage_upload_loop_initial_backoff_ms
Initial backoff interval when there is nothing to upload for a partition, in milliseconds.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 100
cloud_storage_upload_loop_max_backoff_ms
Maximum backoff interval when there is nothing to upload for a partition, in milliseconds.
Unit: milliseconds
Requires restart: No
Visibility: tunable
Type: integer
Accepted values: [-17592186044416
, 17592186044415
]
Default: 10000
cloud_storage_url_style
Specifies the addressing style to use for Amazon S3 requests. This configuration determines how S3 bucket URLs are formatted. Path style is supported for backward compatibility with legacy systems.
When this property is not set (null
), the client tries to use virtual_host
addressing.
If the initial request fails, the client automatically tries the path
style.
If neither addressing style works, Redpanda terminates the startup, requiring manual configuration to proceed.
Requires restart: Yes
Visibility: user
Accepted values:
-
virtual_host
- Example:<bucket-name>.s3.amazonaws.com
-
path
- Example:s3.amazonaws.com/<bucket-name>
-
null
Default: null