Docs Self-Managed Develop Manage Topics Manage Topics Topics provide a way to organize events in a data streaming platform. When you create a topic, the default cluster-level topic configurations are applied using the cluster configuration file, unless you specify different configurations. The following table shows the default cluster-level properties and their equivalent topic-level properties: Cluster property Default Topic property log_cleanup_policy delete cleanup.policy retention_bytes null (no limit) retention.bytes log_retention_ms 604800000 ms (1 week) retention.ms log_segment_ms null (no limit) segment.ms log_segment_size 134217728 bytes (128 MiB) segment.bytes log_compression_type producer compression.type log_message_timestamp_type CreateTime message.timestamp.type kafka_batch_max_bytes 1048576 bytes (1 MiB) max.message.bytes write_caching_default false write.caching These default settings are best suited to a one-broker cluster in a development environment. To learn how to modify the default cluster-wide configurations, see Configure Cluster Properties. Even if you set default values that work for most topics, you may still want to change some properties for a specific topic. For details about topic properties, see Topic Configuration Properties. Create a topic Creating a topic can be as simple as specifying a name for your topic on the command line. For example, to create a topic named xyz, run: rpk topic create xyz This command creates a topic named xyz with one partition and one replica, because these are the default values set in the cluster configuration file. This section shows how to change default settings for a topic. Choose the number of partitions A partition acts as a log file where topic data is written. Dividing topics into partitions allows producers to write messages in parallel and consumers to read messages in parallel. The higher the number of partitions, the greater the throughput. As a general rule, select a number of partitions that corresponds to the maximum number of consumers in any consumer group that will consume the data. For example, suppose you plan to create a consumer group with 10 consumers. To create topic xyz with 10 partitions, run: rpk topic create xyz -p 10 Choose the replication factor Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data. The default replication factor in the cluster configuration is set to 1. By choosing a replication factor greater than 1, you ensure that each partition has a copy of its data on at least one other broker. One replica acts as the leader, and the other replicas are followers. To specify a replication factor of 3 for topic xyz, run: rpk topic create xyz -r 3 The replication factor must be an odd number. Redpanda Data recommends a replication factor of 3 for most use cases. Administrators may set a minimum required replication factor for any new topic in the cluster through the cluster-level minimum_topic_replications property. If you enable Tiered Storage on a topic, you can then use topic recovery to restore data for a deleted topic. Update topic configurations After you create a topic, you can update the topic property settings for all new data written to it. For example, you can add partitions or change the cleanup policy. Add partitions You can assign a certain number of partitions when you create a topic, and add partitions later. For example, suppose you add brokers to your cluster, and you want to take advantage of the additional processing power. To increase the number of partitions for existing topics, run: rpk topic add-partitions [TOPICS...] --num [#] Note that --num <#> is the number of partitions to add, not the total number of partitions. Change the replication factor Suppose you create a topic with the default replication factor of 1 (which is specified in the cluster properties configuration file). Now you want to change the replication factor to 3, so you can have two backups of topic data in case a broker goes down. To set the replication factor to 3, run: rpk topic alter-config [TOPICS...] --set replication.factor=3 The replication factor can’t exceed the number of Redpanda brokers. If you try to set a replication factor greater than the number of brokers, the request is rejected. Change the cleanup policy The cleanup policy determines how to clean up the partition log files when they reach a certain size: delete deletes data based on age or log size. Topics retain all records until then. compact compacts the data by only keeping the latest values for each KEY. compact,delete combines both methods. Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a delete cleanup policy provide a running history of all changes for those topics. For example, to change a topic’s policy to compact, run: rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact For details on compaction in Redpanda, see Compaction settings. Configure write caching Write caching is a relaxed mode of acks=all that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. Write caching applies to user topics. It does not apply to transactions or consumer offsets: data written in the context of a transaction and consumer offset commits is always written to disk and fsynced before being acknowledged to the client. For clusters in development mode, write caching is enabled by default. For clusters in production mode, it is disabled by default. Only enable write caching on workloads that can tolerate some data loss in the case of multiple, simultaneous broker failures. Leaving write caching disabled safeguards your data against complete data center or availability zone failures. Configure at cluster level To enable write caching by default in all user topics, set the cluster-level property write_caching_default: rpk cluster config set write_caching_default=true With write_caching_default set to true at the cluster level, Redpanda fsyncs to disk according to raft_replica_max_pending_flush_bytes and raft_replica_max_flush_delay_ms, whichever is reached first. Configure at topic level To override the cluster-level setting at the topic level, set the topic-level property write.caching: rpk topic alter-config my_topic --set write.caching=true With write.caching enabled at the topic level, Redpanda fsyncs to disk according to flush.ms and flush.bytes, whichever is reached first. Remove a configuration setting You can remove a configuration that overrides the default setting, and the setting will use the default value again. For example, suppose you altered the cleanup policy to use compact instead of the default, delete. Now you want to return the policy setting to the default. To remove the configuration setting cleanup.policy=compact, run rpk topic alter-config with the --delete flag: rpk topic alter-config [TOPICS...] --delete cleanup.policy List topic configuration settings To display all the configuration settings for a topic, run: rpk topic describe <topic-name> -c The -c flag limits the command output to just the topic configurations. This command is useful for checking the default configuration settings before you make any changes and for verifying changes after you make them. The following command output displays after running rpk topic describe test-topic, where test-topic was created with default settings: rpk topic describe test_topic SUMMARY ======= NAME test_topic PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG max.message.bytes 1048576 DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.delete true DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG Suppose you add two partitions, and increase the number of replicas to 3. The new command output confirms the changes in the SUMMARY section: SUMMARY ======= NAME test_topic PARTITIONS 3 REPLICAS 3 Delete a topic To delete a topic, run: rpk topic delete <topic-name> When a topic is deleted, its underlying data is deleted, too. To delete multiple topics at a time, provide a space-separated list. For example, to delete two topics named topic1 and topic2, run: rpk topic delete topic1 topic2 You can also use the -r flag to specify one or more regular expressions; then, any topic names that match the pattern you specify are deleted. For example, to delete topics with names that start with “f” and end with “r”, run: rpk topic delete -r '^f.*' '.*r$' Note that the first regular expression must start with the ^ symbol, and the last expression must end with the $ symbol. This requirement helps prevent accidental deletions. Delete records from a topic Redpanda lets you delete data from the beginning of a partition up to a specific event, also known as offset. The offset represents the true creation time of the event, not the time when it was stored by Redpanda. Deleting records frees up disk space, which is especially helpful if your producers are pushing more data than you anticipated in your retention plan. Do this when you know that all consumers have read up to that given offset, and the data is no longer needed. There are different ways to delete records from a topic, including using the rpk topic trim-prefix command or using the DeleteRecords Kafka API with Kafka clients. To delete records, cleanup.policy must be set to delete or compact,delete. Object storage is deleted asynchronously. After messages are deleted, the partition’s start offset will have advanced, but garbage collection of deleted segments may not be complete. Similar to Kafka, after deleting records, local storage and object storage may still contain data for deleted offsets. (Redpanda does not truncate segments. Instead, it bumps the start offset, then it attempts to delete as many whole segments as possible.) Data before the new start offset is not visible to clients but could be read by someone with access to the local disk of a Redpanda node. Next steps Configure Producers Suggested labs Stream Stock Market Data from a CSV file Using Node.jsStream Stock Market Data from a CSV file Using PythonBuild a Chat Room Application with Redpanda and JavaBuild a Chat Room Application with Redpanda and GolangBuild a Chat Room Application with Redpanda and RustBuild a Chat Room Application with Redpanda and Node.jsBuild a Chat Room Application with Redpanda and PythonSee moreSearch all labs Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Use Redpanda with the HTTP Proxy API Produce Data