Tiered Storage

Loading...

important

This feature requires an Enterprise license. To upgrade, contact Redpanda sales.

Tiered Storage helps to lower storage costs by offloading log segments to object storage. You can specify the amount of storage you want to retain in local storage. You don't need to specify which log segments you want to move, because Redpanda moves them automatically based on cluster-level configuration properties. Tiered Storage indexes where data is offloaded, so it can look up the data when you need it.

The following image illustrates the Tiered Storage architecture: remote write uploads data from Redpanda to object storage, and remote read fetches data from object storage to Redpanda.

caution

Do not use Tiered Storage with data transforms, which is a lab feature in technical preview. Lab features are not supported for production usage.

Prerequisites

This feature requires an Enterprise license. To upgrade, contact Redpanda sales.

Set up Tiered Storage

To set up Tiered Storage:

1. Configure object storage.
2. Enable Tiered Storage. You can enable Tiered Storage for the cluster (all topics) or for specific topics.
3. Set retention limits.

Configure object storage

Redpanda natively supports Tiered Storage with Amazon Simple Storage Service (S3), Google Cloud Storage (GCS, which uses the Google Cloud Platform S3 API), and Microsoft Azure Blob Storage (ABS).

important

• Migrating topics from one object storage provider to another is not supported.
• Migrating topics from one bucket or container to another is not supported.

Amazon S3

Configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.

To configure access to an S3 bucket with an IAM role:

1. Configure an IAM role.

2. Run the rpk cluster config edit command, then edit the following required properties:

   cloud_storage_enabled: true
   cloud_storage_credentials_source: aws_instance_metadata
   cloud_storage_region: <region>
   cloud_storage_bucket: <redpanda_bucket_name>
   notes

   • Replace <placeholders> with your own values.
   • For additional properties, see Tiered Storage configuration properties.
   important

   Do not set an object storage property to an empty string "" or to null as a way to reset it to its default value. To reset a property to its default value, run rpk cluster config force-reset <config_name> or remove that line from the cluster configuration with rpk cluster config edit.

To configure access to an S3 bucket with access keys instead of an IAM role:

1. Grant a user the following permissions to read and create objects on the bucket to be used with the cluster (or on all buckets): GetObject, DeleteObject, PutObject, PutObjectTagging, ListBucket.

2. Copy the access key and secret key for the cloud_storage_access_key and cloud_storage_secret_key cluster properties.

3. Run the rpk cluster config edit command, then edit the following required properties:

   cloud_storage_enabled: true
   cloud_storage_access_key: <access_key>
   cloud_storage_secret_key: <secret_key>
   cloud_storage_region: <region>
   cloud_storage_bucket: <redpanda_bucket_name>
   notes

   • Replace <placeholders> with your own values.
   • For additional properties, see Tiered Storage configuration properties.
   important

   Do not set an object storage property to an empty string "" or to null as a way to reset it to its default value. To reset a property to its default value, run rpk cluster config force-reset <config_name> or remove that line from the cluster configuration with rpk cluster config edit.

Google Cloud Storage

Configure access to Google Cloud Storage with either an IAM role attached to the instance or with access keys.

To configure access to Google Cloud Storage with an IAM role:

1. Configure an IAM role.

2. Run the rpk cluster config edit command, then edit the following required properties:

   cloud_storage_enabled: true
   cloud_storage_credentials_source: gcp_instance_metadata
   cloud_storage_region: <region>
   cloud_storage_bucket: <redpanda_bucket_name>
   notes

   • Replace <placeholders> with your own values.
   • For additional properties, see Tiered Storage configuration properties.
   important

   Do not set an object storage property to an empty string "" or to null as a way to reset it to its default value. To reset a property to its default value, run rpk cluster config force-reset <config_name> or remove that line from the cluster configuration with rpk cluster config edit.

To configure access to Google Cloud Storage with access keys instead of an IAM role:

1. Choose a uniform access control when you create the bucket.

2. Use a Google-managed encryption key.

3. Set a default project.

4. Create a service user with HMAC keys and copy the access key and secret key for the cloud_storage_access_key and cloud_storage_secret_key cluster properties.

5. Run the rpk cluster config edit command, then edit the following required properties:

   cloud_storage_enabled: true
   cloud_storage_api_endpoint: storage.googleapis.com
   cloud_storage_access_key: <access_key>
   cloud_storage_secret_key: <secret_key>
   cloud_storage_bucket: <redpanda_bucket_name>
   cloud_storage_region: <region>
   notes

   • Replace <placeholders> with your own values.
   • For additional properties, see Tiered Storage configuration properties.
   important

   Do not set an object storage property to an empty string "" or to null as a way to reset it to its default value. To reset a property to its default value, run rpk cluster config force-reset <config_name> or remove that line from the cluster configuration with rpk cluster config edit.

Azure Blob Storage

To configure access to Azure Blob Storage with shared keys:

1. Copy an account access key for the Azure container you want Redpanda to use and enter it in the cloud_storage_azure_shared_key property. For information on how to view your account access keys, see the Azure documentation.

2. Run the rpk cluster config edit command, then edit the following required properties:

   cloud_storage_enabled: true
   cloud_storage_azure_shared_key: <azure_account_access_key>
   cloud_storage_azure_storage_account: <azure_account_name>
   cloud_storage_azure_container: <redpanda_container_name>
   notes

   • Replace <placeholders> with your own values.
   • For additional properties, see Tiered Storage configuration properties.
   • For information about how to grant access from an internet IP range (if you need to open additional routes/ports between your broker nodes and Azure Blob Storage; for example, in a hybrid cloud deployment), see the Microsoft documentation.
   • For information about shared key authentication, see the Microsoft documentation.
   important

   Do not set an object storage property to an empty string "" or to null as a way to reset it to its default value. To reset a property to its default value, run rpk cluster config force-reset <config_name> or remove that line from the cluster configuration with rpk cluster config edit.

Enable Tiered Storage

1. To enable Tiered Storage, set cloud_storage_enabled to true.

2. Configure topics for Tiered Storage. You can configure either all topics in a cluster or only specific topics.

   • Enable Tiered Storage for a cluster
   • Enable Tiered Storage for specific topics

Enable Tiered Storage for a cluster

To enable Tiered Storage for a cluster (in addition to setting cloud_storage_enabled to true), set the following cluster-level properties to true:

• cloud_storage_enable_remote_write
• cloud_storage_enable_remote_read

When you enable Tiered Storage for a cluster, you enable it for all existing topics in the cluster. When cluster-level properties are changed, the changes apply only to new topics, not existing topics.

note

The cloud_storage_enable_remote_write and cloud_storage_enable_remote_read cluster-level properties are essentially creation-time defaults for the redpanda.remote.write and redpanda.remote.read topic-level properties.

Enable Tiered Storage for specific topics

To enable Tiered Storage for a new or existing topic (in addition to setting cloud_storage_enabled to true), set the following topic-level properties to true:

• redpanda.remote.write
• redpanda.remote.read

For example, to create a new topic with Tiered Storage:

rpk topic create <topic_name> -c redpanda.remote.read=true -c redpanda.remote.write=true

To enable Tiered Storage on an existing topic, run:

rpk topic alter-config <topic_name> --set redpanda.remote.read=true --set redpanda.remote.write=true

Topic-level properties override cluster-level properties. For example, for new topics, if cloud_storage_enable_remote_write is set to true, you can set redpanda.remote.write to false to turn it off for a particular topic.

Tiered Storage topic-level properties:

Property	Description
redpanda.remote.write	Uploads data from Redpanda to object storage. Overrides the cluster-level cloud_storage_enable_remote_write configuration for the topic.
redpanda.remote.read	Fetches data from object storage to Redpanda. Overrides the cluster-level cloud_storage_enable_remote_read configuration for the topic.
redpanda.remote.recovery	Recovers or reproduces a topic from object storage. Use this property during topic creation. It does not apply to existing topics.
redpanda.remote.delete	When set to true, deleting a topic also deletes its objects in object storage. Both redpanda.remote.write and redpanda.remote.read must be enabled, and the topic must not be a Remote Read Replica topic. When set to false, deleting a topic does not delete its objects in object storage. Default is true for new topics.

The following tables list outcomes for combinations of cluster-level and topic-level configurations:

Cluster-level configuration with cloud_storage_enable_remote_write	Topic-level configuration with redpanda.remote.write	Outcome: whether remote write is enabled or disabled on the topic
true	Not set	Enabled
true	false	Disabled
true	true	Enabled
false	Not set	Disabled
false	false	Disabled
false	true	Enabled

Cluster-level configuration with cloud_storage_enable_remote_read	Topic-level configuration with redpanda.remote.read	Outcome: whether remote read is enabled or disabled on the topic
true	Not set	Enabled
true	false	Disabled
true	true	Enabled
false	Not set	Disabled
false	false	Disabled
false	true	Enabled

Set retention limits

Redpanda supports retention limits and compaction for topics using Tiered Storage. Set retention limits to purge topic data after it reaches a specified age or size.

Starting in Redpanda version 22.3, object storage is the default storage tier for all streaming data, and retention properties work the same for Tiered Storage topics and local storage topics. Data is retained in object storage until it reaches the configured time or size limit.

note

During upgrade, Redpanda preserves retention settings for existing topics.

Data becomes eligible for deletion from object storage following retention.ms or retention.bytes. For example, if retention.bytes is set to 10 GiB, then every partition in the topic has a limit of 10 GiB in object storage. When retention.bytes is exceeded by data in object storage, the data in object storage is trimmed. If neither retention.ms nor retention.bytes is specified, then cluster-level defaults are used. For more information, see Configure message retention.

Compacted topics in Tiered Storage

When you set cleanup.policy=compact for a topic, nothing gets deleted from object storage.

For compacted topics, Redpanda compacts segments after they have been uploaded to object storage. Redpanda initially uploads all uncompacted segments. It then re-uploads the segments with compaction applied. It's likely that some segments in object storage are not compacted, but the Tiered Storage read path can manage this.

Manage local capacity for Tiered Storage topics

You can use properties to control retention of topic data in local storage. With Tiered Storage enabled, data in local storage expires after the topic-level properties retention.local.target.ms or retention.local.target.bytes. (These settings are equivalent to retention.ms and retention.bytes without Tiered Storage.)

You can also use the cluster-level properties retention_local_target_ms_default and retention_local_target_bytes_default. Settings can depend on the size of your drive, how many partitions you have, and how much data you keep for each partition.

When set, Redpanda keeps actively-used and sequential (next segment) data in local cache and targets to maintain this age of data in local storage. It purges data based on actual available local volume space, without forcing disk full situations when there is data skew.

At topic creation with Tiered Storage enabled:

• If retention.ms or retention.bytes is set, Redpanda initializes the retention.local.target.* properties.
• If retention.local.target.ms or retention.local.target.bytes is set, Redpanda initializes the min(retention.bytes, retention.local.target.bytes) and max(retention.ms, retention.local.target.ms) properties.
• If properties are not specified:
  • Starting in version 22.3, new topics use the default retention values of one day for local storage (retention_local_target_ms_default) and seven days for all storage, including object storage (delete_retention_ms).
  • Upgraded topics retain their historical defaults of infinite retention.

After topic configuration, if Tiered Storage was disabled and must be enabled, or was enabled and must be disabled, Redpanda uses the local retention properties set for the topic.

Remote write

Remote write is the process that constantly uploads log segments to object storage. The process is created for each partition and runs on the leader broker of the partition. It only uploads the segments that contain offsets that are smaller than the last stable offset. This is the latest offset that the client can read.

To ensure all data is uploaded, you must enable remote write before any data is produced to the topic. If remote write is not enabled, data may be deleted due to retention settings.

To enable Tiered Storage, use remote write and remote read.

note

If you only enable remote write on a topic, you have a simple backup that you can use for recovery. See Data Archiving.

To create a topic with remote write enabled:

rpk topic create <topic_name> -c redpanda.remote.write=true

To enable remote write on an existing topic:

rpk topic alter-config <topic_name> --set redpanda.remote.write=true

If remote write is enabled, log segments are not deleted until they’re uploaded to object storage. Because of this, the log segments may exceed the configured retention period until they’re uploaded, so the topic might require more disk space. This prevents data loss if segments cannot be uploaded fast enough or if the retention period is very short.

tip

A constant stream of data is necessary to build up the log segments and roll them into object storage. To see new log segments faster, you can edit the segment.bytes topic-level property. Or, you can edit the cloud_storage_segment_max_upload_interval_sec cluster-level property, which sets the frequency with which new segments are generated in Tiered Storage, even if the local segment has not exceeded the segment.bytes size or the segment.ms age.

note

Starting in version 22.3, when you delete a topic in Redpanda, the data is also deleted in object storage. See Enable Tiered Storage for a topic.

Idle timeout

You can configure Redpanda to start a remote write periodically. This is useful if the ingestion rate is low and the segments are kept open for long periods of time. You specify a number of seconds for the timeout, and if that time has passed since the previous write and the partition has new data, Redpanda starts a new write. This limits data loss in the event of catastrophic failure and adds a guarantee that you only lose the specified number of seconds of data.

Setting idle timeout to a very short interval can create a lot of small files, which can affect throughput. If you decide to set a value for idle timeout, start with 600 seconds, which prevents the creation of so many small files that throughput is affected when you recover the files.

Use the cloud_storage_segment_max_upload_interval_sec property to set the number of seconds for idle timeout. If this property is empty, Redpanda uploads metadata to object storage, but the segment is not uploaded until it reaches the segment.bytes size. By default, the property is null.

Reconciliation

Reconciliation is a Redpanda process that monitors partitions and decides which partitions are uploaded on each broker to guarantee that data is uploaded only once. It runs periodically on every broker. It also balances the workload evenly between brokers.

note

The broker uploading to object storage is always with the partition leader. Therefore, when partition leadership balancing occurs, Redpanda stops uploads to object storage from one broker and starts uploads on another broker.

Upload controller

Remote write uses a proportional derivative (PD) controller to minimize the backlog size for the write. The backlog consists of data that has not been uploaded to object storage but must be uploaded eventually.

The upload controller prevents Redpanda from running out of disk space. If remote.write is set to true, Redpanda cannot evict log segments that have not been uploaded to object storage. If the remote write process cannot keep up with the amount of data that needs to be uploaded to object storage, the upload controller increases priority for the upload process. The upload controller measures the size of the upload periodically and tunes the priority of the remote write process.

Remote read

Remote read fetches data from object storage using the Kafka API. Use remote read in conjunction with remote write to enable Tiered Storage.

In general, when data is evicted locally, it is no longer available. If the consumer starts consuming the partition from the beginning, the first offset is the smallest offset available locally. However, if Tiered Storage is enabled with the redpanda.remote.read and redpanda.remote.write properties, the data is always uploaded to object storage before it's deleted. This guarantees that the data is always available either locally or remotely.

When data is available remotely and Tiered Storage is enabled, the client can start consuming data, even if the data is no longer stored locally.

To create a topic with remote read enabled:

rpk topic create <topic_name> -c -c redpanda.remote.read=true

To enable remote read on an existing topic:

rpk topic alter-config <topic_name> --set redpanda.remote.read=true

See also: Remote Read Replicas

Caching

When the Kafka client fetches an offset range that isn’t available locally in the Redpanda data directory, Redpanda downloads remote segments from object storage. These downloaded segments are stored in the object storage cache directory. The cache directory is created in the Redpanda data directory by default, but it can be placed anywhere in the system. For example, you might want to put the cache directory to a dedicated drive with cheaper storage.

If you don’t specify a cache location in the redpanda.yaml file, the cache directory is created in <redpanda_data_directory>/cloud_storage_cache.

Use the cloud_storage_cache_directory property on each broker to specify a different location for the cache directory. You must specify the full path.

Redpanda checks the cache periodically, and if the size of the stored data is larger than the configured limit, the eviction process starts. The eviction process removes segments that haven’t been accessed recently, until the size of the cache drops 20%.

Use the following cluster-level properties to set the maximum cache size and cache check interval. Edit them with the rpk cluster config edit command.

Property	Description
cloud_storage_cache_size	Maximum size of the disk cache used by Tiered Storage. Default is 20 GiB.
cloud_storage_cache_check_interval	The time, in milliseconds, between cache checks. The size of the cache can grow quickly, so it’s important to have a small interval between checks. However, if the checks are too frequent, they consume a lot of resources. Default is 30000 ms.

Disaster recovery

When you create a topic, you can use remote recovery to download the topic data from object storage. You can also use remote recovery to restore a topic that was deleted from a cluster.

caution

Remote recovery is not intended to be a fully-functional disaster recovery solution. It is only safe when no other clusters are still writing to the specified bucket or container. Recovery does not provide snapshot-style consistency: some partitions in some topics will be more up to date than others. For example, restoring a consumer offsets topic could have offsets that point ahead of the restored data.

To create a new topic using remote recovery:

rpk topic create <topic_name> -c redpanda.remote.recovery=true

To create a new topic using remote recovery and enable Tiered Storage on the new topic with the redpanda.remote.write and redpanda.remote.read properties:

rpk topic create <topic_name> -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true

Retries and backoff

If the object storage provider replies with an error message that the server is busy, Redpanda retries the request. Redpanda may retry on other errors, depending on the object storage provider.

Redpanda always uses exponential backoff with cloud connections. You can configure the cloud_storage_initial_backoff_ms property to set the time used as an initial backoff interval in the exponential backoff algorithm to handle an error. Default is 100 ms.

Transport

Tiered Storage creates a connection pool for each CPU that limits simultaneous connections to the object storage provider. It also uses persistent HTTP connections with a configurable maximum idle time. A custom S3 client is used to send and receive data.

For normal usage, you do not need to configure the transport properties. The Redpanda defaults are sufficient, and the certificates used to connect to the object storage client are available through public key infrastructure. Redpanda detects the location of the CA certificates automatically.

Redpanda uses the following properties to configure transport.

Property	Description
cloud_storage_max_connections	The maximum number of connections to object storage on a broker for each CPU. Remote read and remote write share the same pool of connections. This means that if a connection is used to upload a segment, it cannot be used to download another segment. If this value is too small, some workloads might starve for connections, which results in delayed uploads and downloads. If this value is too large, Redpanda tries to upload a lot of files at the same time and might overwhelm the system. Default is 20.
cloud_storage_segment_upload_timeout_ms	Timeout for segment upload. Redpanda retries the upload after the timeout. Default is 30000 ms.
cloud_storage_manifest_upload_timeout_ms	Timeout for manifest upload. Redpanda retries the upload after the timeout. Default is 10000 ms.
cloud_storage_max_connection_idle_time_ms	The maximum idle time for persistent HTTP connections. This differs depending on the object storage provider. Default is 5000 ms, which is sufficient for most providers.
cloud_storage_segment_max_upload_interval_sec	The number of seconds for idle timeout. If this property is empty, Redpanda uploads metadata to the object storage, but the segment is not uploaded until it reaches the segment.bytes size. By default, the property is empty.
cloud_storage_trust_file	The public certificate used to validate the TLS connection to object storage. If this is empty, Redpanda uses your operating system's CA cert pool.

Object storage housekeeping

To improve performance and scalability, Redpanda performs object storage housekeeping when the system is idle. This housekeeping includes adjacent segment merging, which analyzes the data layout of the partition in object storage. If it finds a run of small segments, it can merge and reupload the segment.

To determine when the system is idle for housekeeping, Redpanda constantly calculates object storage utilization using the moving average with a sliding window algorithm. The width of the sliding window is defined by the cloud_storage_idle_timeout_ms property, which has a default of 10 seconds. If the utilization (requests per second) drops below the threshold, then object storage is considered idle, and object storage housekeeping begins. The threshold is defined by the cloud_storage_idle_threshold_rps property, which has a default of one request per second. Object storage is considered idle if, during the last 10 seconds, there were 10 or less object storage API requests.

If the object storage API becomes active after housekeeping begins, then housekeeping is paused until it becomes idle again. If object storage is not idle for cloud_storage_housekeeping_interval_ms, then housekeeping is forced to run until it completes. This guarantees that all housekeeping jobs are run once for every cloud_storage_housekeeping_interval_ms.

Adjacent segment merging

By default, and as part of this object storage housekeeping, Redpanda runs adjacent segment merging on all segments in object storage that are smaller than the threshold. Two properties control the behavior of cloud_storage_enable_segment_merging:

• cloud_storage_segment_size_target: The desired segment size in object storage. The default segment size is the local segment size for the topic, which is controlled by the log_segment_size configuration property and the segment.bytes topic property. You can set the cloud_storage_segment_size_target property to a value larger than the default segment size, but because this triggers a lot of segment re-uploads, it's not recommended.
• cloud_storage_segment_size_min: The smallest segment size that Redpanda keeps in object storage. The default is 50% of log_segment_size.

If the adjacent segment merging job finds a run of small segments, it can perform one of the following operations:

• Merge and re-upload a segment with a size up to cloud_storage_segment_size_target.
• Merge and re-upload a segment with a size less than cloud_storage_segment_size_min if there are no other options (the run of small segments is followed by the large segment).
• Wait until new segments are added if the run is at the end of the partition.

Suppose the cloud_storage_segment_max_upload_interval_sec property is set and the partition contains a large number of small segments. For example, if cloud_storage_segment_max_upload_interval_ms is set to 10 minutes and the produce rate is 1 MiB per minute, then Redpanda uploads a new 10 MiB segment every 10 minutes. If adjacent segment merging is enabled and cloud_storage_segment_size_target is set to 500 MiB, then every 50 segments are re-uploaded as one large 500 MiB segment. This doubles the amount of data that Redpanda uploads to object storage, but it also reduces the memory footprint of the partition, which results in better scalability because 98% less memory is needed to keep information about the uploaded segment.

note

Adjacent segment merging doesn't work for compacted topics, because compacted segments are reuploaded after they're compacted. The results are the same.

Tiered Storage configuration properties

The following list contains cluster-level configuration properties for Tiered Storage. Configure or verify the following properties before you use Tiered Storage:

Property	Description
cloud_storage_enabled	Global property that enables Tiered Storage. Set to true to enable Tiered Storage. Default is false.
cloud_storage_region	Object storage region. Required for AWS and GCS.
cloud_storage_bucket	AWS or GCS bucket name. Required for AWS and GCS.
cloud_storage_credentials_source	AWS or GCS instance metadata. Required for AWS and GCS authentication with IAM roles.
cloud_storage_access_key	AWS or GCS access key. Required for AWS and GCS authentication with access keys.
cloud_storage_secret_key	AWS or GCS secret key. Required for AWS and GCS authentication with access keys.
cloud_storage_api_endpoint	AWS or GCS API endpoint. - For AWS, this can be left blank. It’s generated automatically using the region and bucket. - For GCS, use storage.googleapis.com.
cloud_storage_azure_container	Azure container name. Required for ABS.
cloud_storage_azure_storage_account	Azure account name. Required for ABS.
cloud_storage_azure_shared_key	Azure shared key. Required for ABS.
cloud_storage_cache_size	Maximum size of the disk cache used by Tiered Storage. Default is 20 GiB.

In addition, you might want to change the following node property for each broker:

Property	Description
cloud_storage_cache_directory	The directory for the Tiered Storage cache. You must specify the full path. Default is: <redpanda-data-directory>/cloud_storage_cache.

You may want to configure the following properties:

Property	Description
cloud_storage_max_connections	The maximum number of connections to object storage on a broker for each CPU. Remote read and remote write share the same pool of connections. This means that if a connection is used to upload a segment, it cannot be used to download another segment. If this value is too small, some workloads might starve for connections, which results in delayed uploads and downloads. If this value is too large, Redpanda tries to upload a lot of files at the same time and might overwhelm the system. Default is 20.
cloud_storage_initial_backoff_ms	The time, in milliseconds, for an initial backoff interval in the exponential backoff algorithm to handle an error. Default is 100 ms.
cloud_storage_segment_upload_timeout_ms	Timeout for segment upload. Redpanda retries the upload after the timeout. Default is 30000 ms.
cloud_storage_manifest_upload_timeout_ms	Timeout for manifest upload. Redpanda retries the upload after the timeout. Default is 10000 ms.
cloud_storage_max_connection_idle_time_ms	The maximum idle time for persistent HTTP connections. Differs depending on the object storage provider. Default is 5000 ms, which is sufficient for most providers.
cloud_storage_segment_max_upload_interval_sec	Sets the number of seconds for idle timeout. If this property is empty, Redpanda uploads metadata to the object storage, but the segment is not uploaded until it reaches the segment.bytes size. By default, the property is empty.
cloud_storage_cache_check_interval	The time, in milliseconds, between cache checks. The size of the cache can grow quickly, so it’s important to have a small interval between checks, but if the checks are too frequent, they consume a lot of resources. Default is 30000 ms.
cloud_storage_idle_timeout_ms	The width of the sliding window for the moving average algorithm that calculates object storage utilization. Default is 10 seconds.
cloud_storage_idle_threshold_rps	The utilization threshold for object storage housekeeping. Object storage is considered idle if, during the last 10 seconds, there were 10 or less object storage API requests. Default is 1 request per second.
cloud_storage_enable_segment_merging	Enables adjacent segment merging on all segments in object storage that are smaller than the threshold. Two properties control this behavior: cloud_storage_segment_size_target and cloud_storage_segment_size_min. Default is enabled.
cloud_storage_segment_size_target	The desired segment size in object storage. The default segment size is controlled by log_segment_size and the segment.bytes topic configuration property. This property can be set to a value larger than this default segment size, but because that triggers a lot of segment reuploads, it's not recommended.
cloud_storage_segment_size_min	The smallest segment size in object storage that Redpanda keeps. Default is 50% of log segment size.

Under normal circumstances, you should not need to configure the following tunable properties:

Property	Description
cloud_storage_upload_ctrl_update_interval_ms	The recompute interval for the upload controller. Default is 60000 ms.
cloud_storage_upload_ctrl_p_coeff	The proportional coefficient for the upload controller. Default is -2.
cloud_storage_upload_ctrl_d_coeff	The derivative coefficient for the upload controller. Default is 0.
cloud_storage_upload_ctrl_min_shares	The minimum number of I/O and CPU shares that the remote write process can use. Default is 100.
cloud_storage_upload_ctrl_max_shares	The maximum number of I/O and CPU shares that the remote write process can use. Default is 1000.
cloud_storage_disable_tls	Disables TLS encryption. Set to true if TLS termination is done by the proxy, such as HAProxy. Default is false.
cloud_storage_api_endpoint_port	Overrides the default API endpoint port. Default is 443.
cloud_storage_trust_file	The public certificate used to validate the TLS connection to object storage. If this is empty, Redpanda uses your operating system's CA cert pool.
cloud_storage_reconciliation_interval_ms	Deprecated. The interval, in milliseconds, to reconcile partitions that need to be uploaded. A long reconciliation interval can result in a delayed reaction to topic creation, topic deletion, or leadership rebalancing events. A short reconciliation interval guarantees that new partitions are picked up quickly, but the process uses more resources. Default is 10000 ms.

Suggested reading

• How we built shadow indexing, the subsystem powering Tiered Storage
• Configure Cluster Properties