# Manage Topics with the Redpanda Operator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Topics with the Redpanda Operator latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "true" page-eol-date: April 30, 2025 latest-console-tag: v3.7.3 latest-connect-version: 4.93.0 docname: kubernetes/k-manage-topics page-component-name: streaming page-version: "24.1" page-component-version: "24.1" page-component-title: Streaming page-relative-src-path: kubernetes/k-manage-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/v/24.1/modules/manage/pages/kubernetes/k-manage-topics.adoc description: Use the Topic resource to declaratively create Kafka topics as part of a Redpanda deployment. Each Topic resource is mapped to a topic in your Redpanda cluster. The topic controller keeps the corresponding Kafka topic in sync with the Topic resource. page-git-created-date: "2024-01-04" page-git-modified-date: "2024-06-28" support-status: past end-of-life --- The Redpanda Operator allows you to declaratively create and manage Kafka topics using [Topic custom resources](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/) (resources) in Kubernetes. Each Topic resource is mapped to a topic in your Redpanda cluster. The topic controller, a component of the Redpanda Operator, keeps the corresponding Kafka topic in sync with the Topic resource. This resource allows you to create topics as part of a Redpanda deployment. ## [](#prerequisites)Prerequisites You must have the following: - Kubernetes cluster: Ensure you have a running Kubernetes cluster, either locally, such as with minikube or kind, or remotely. - [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl): Ensure you have the `kubectl` command-line tool installed and configured to communicate with your cluster. - Redpanda: Ensure you have the [Redpanda Operator and a Redpanda resource deployed](https://docs.redpanda.com/streaming/24.1/deploy/deployment-option/self-hosted/kubernetes/k-production-deployment/) in your Kubernetes cluster. ## [](#limitations)Limitations You cannot create access control lists (ACLs) directly in the Topic resource. To create ACLs for your topics, you must use `rpk` or another Kafka client. For details about ACLs, see [Redpanda Authorization Mechanisms](https://docs.redpanda.com/streaming/24.1/manage/security/authorization/). ## [](#create-a-topic)Create a topic You can create a topic using a Topic resource: ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Topic metadata: name: spec: kafkaApiSpec: brokers: [] tls: {} sasl: {} partitions: replicationFactor: additionalConfig: {} metricsNamespace: overwriteTopicName: interval: ``` - [`metadata.name`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec) (**required**): The name of the Topic resource. If the [`overwriteTopicName`](#overwrite) property is not set, the name of the Topic resource is also given to the topic in Redpanda. Valid names must consist of lowercase alphanumeric characters, hyphens (-), or periods (.). Names cannot start or end with a non-alphanumeric character. Underscores (\_) are not allowed. For example, `chat-room` is a valid name, whereas `chat_room` is not. To use other characters such as underscores in your topic names, use the [`overwriteTopicName`](#overwrite) property. - [`spec.kafkaApiSpec`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-kafkaapispec) (**required**): Configuration details for connecting to Redpanda brokers. - [`spec.partitions`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec): The number of topic shards distributed across the brokers in a Redpanda cluster. This value cannot be decreased post-creation. Overrides the default cluster property [`default_topic_partitions`](https://docs.redpanda.com/streaming/24.1/reference/properties/cluster-properties/#default_topic_partitions). - [`spec.replicationFactor`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec): Specifies the number of topic replicas. The value must be an odd number. Overrides the default cluster property [`default_topic_replications`](https://docs.redpanda.com/streaming/24.1/reference/properties/cluster-properties/#default_topic_replications). - [`spec.additionalConfig`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec): A map of any topic-specific configuration options. See [Topic Configuration Properties](https://docs.redpanda.com/streaming/24.1/reference/properties/topic-properties/). - [`spec.metricsNamespace`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec): The fully-qualified name of the topic metrics for use in multi-operator environments. Defaults to `redpanda-operator`. - [`spec.overwriteTopicName`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec): Overwrites the topic name in `metadata.name`. - [`spec.interval`](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/#k8s-api-github-com-redpanda-data-redpanda-src-go-k8s-apis-cluster-redpanda-com-v1alpha1-topicspec): Sets the reconciliation interval for the topic controller. Default is 3 seconds (`3s`). The default settings are best suited to a one-broker cluster in a development environment. To learn how to modify the default values in the configuration file, see [Configure Cluster Properties](https://docs.redpanda.com/streaming/24.1/manage/cluster-maintenance/cluster-property-configuration/). Even if you set default values that work for most topics, you may still want to change some properties for a specific topic. ### [](#example-topic-resource)Example Topic resource This example creates a topic called `chat-room`. The `kafkaApiSpec` property configures the connection to a Redpanda cluster using the defaults in the Redpanda Helm chart. `example-topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Topic metadata: name: chat-room spec: partitions: 3 replicationFactor: 3 additionalConfig: cleanup.policy: "compact" kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" ``` ```bash kubectl apply -f example-topic.yaml --namespace ``` Replace `` with the namespace in which you deployed Redpanda. ### [](#choose-the-number-of-partitions)Choose the number of partitions A partition acts as a log file where topic data is written. Dividing topics into partitions allows producers to write messages in parallel and consumers to read messages in parallel. The higher the number of partitions, the greater the throughput. > 💡 **TIP** > > As a general rule, select a number of partitions that corresponds to the maximum number of consumers in any consumer group that will consume the data. ### [](#choose-the-replication-factor)Choose the replication factor Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data. The default replication factor in the cluster configuration is set to 1. By choosing a replication factor greater than 1, you ensure that each partition has a copy of its data on at least one other broker. One replica acts as the leader, and the other replicas are followers. > 📝 **NOTE** > > The replication factor must be an odd number. Redpanda Data recommends a replication factor of 3 for most use cases. ### [](#choose-the-cleanup-policy)Choose the cleanup policy The cleanup policy determines how to manage the partition log files when they reach a certain size: - `delete` deletes data based on age or log size. - `compact` compacts the data by only keeping the latest values for each KEY. - `compact,delete` combines both methods. If the cleanup policy is `delete` or `compact,delete` you can [Delete records from a topic](#delete-records-from-a-topic). ### [](#configure-write-caching)Configure write caching Write caching is a relaxed mode of [`acks=all`](https://docs.redpanda.com/streaming/24.1/develop/produce-data/configure-producers/#acksall) that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. Write caching applies to user topics. It does not apply to transactions or consumer offsets. Transactions and offset commits are always fsynced. > 📝 **NOTE** > > For clusters in [development mode](https://docs.redpanda.com/streaming/24.1/reference/rpk/rpk-redpanda/rpk-redpanda-mode/#development-mode), write caching is enabled by default. For clusters in production mode, it is disabled by default. Only enable write caching on workloads that can tolerate some data loss in the case of multiple, simultaneous broker failures. Leaving write caching disabled safeguards your data against complete data center or availability zone failures. #### [](#configure-at-cluster-level)Configure at cluster level To enable write caching by default in all user topics, set the cluster-level property [`write_caching_default`](https://docs.redpanda.com/streaming/24.1/reference/properties/cluster-properties/#write_caching_default): ##### Helm + Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: config: cluster: write_caching_default: true ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` ##### Helm ###### --values `write-caching.yaml` ```yaml config: cluster: write_caching_default: true ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values write-caching.yaml --reuse-values ``` ###### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set config.cluster.write_caching_default=true ``` With `write_caching` enabled at the cluster level, Redpanda fsyncs to disk according to [`raft_replica_max_pending_flush_bytes`](https://docs.redpanda.com/streaming/24.1/reference/properties/cluster-properties/#raft_replica_max_pending_flush_bytes) and [`raft_replica_max_flush_delay_ms`](https://docs.redpanda.com/streaming/24.1/reference/properties/cluster-properties/#raft_replica_max_flush_delay_ms), whichever is reached first. #### [](#configure-at-topic-level)Configure at topic level To override the cluster-level setting at the topic level, set the topic-level property [`write.caching`](https://docs.redpanda.com/streaming/24.1/reference/properties/topic-properties/#writecaching): `example-topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Topic metadata: name: chat-room spec: partitions: 3 replicationFactor: 3 additionalConfig: write.caching: true kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" ``` With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to [`flush.ms`](https://docs.redpanda.com/streaming/24.1/reference/properties/topic-properties/#flushms) and [`flush.bytes`](https://docs.redpanda.com/streaming/24.1/reference/properties/topic-properties/#flushbytes), whichever is reached first. ## [](#verify-a-topic)Verify a topic After deploying a Topic resource, verify that the Redpanda Operator reconciled it. ```bash kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace ``` Example output ```json { "level":"info", "ts":"2023-09-25T16:20:09.538Z", "logger":"TopicReconciler.Reconcile", "msg":"Starting reconcile loop", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"chat-room", "namespace":"" }, "namespace":"", "name":"chat-room", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432" } { "level":"info", "ts":"2023-09-25T16:20:09.581Z", "logger":"TopicReconciler.Reconcile", "msg":"reconciliation finished in 43.436125ms, next run in 3s", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"chat-room", "namespace":"" }, "namespace":"", "name":"chat-room", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432", "result": { "Requeue":false, "RequeueAfter":3000000000 } } ``` Then, verify that the topic now exists in your Redpanda cluster: ```bash kubectl --namespace exec -ti -c -- \ rpk topic list ``` Example output: NAME PARTITIONS REPLICAS chat-room 3 3 ## [](#view-topic-configuration)View topic configuration To view the configuration for all Topic resources: ```bash kubectl get topic -o yaml --namespace ``` You should see a list of all Topic resources and their configurations. Example output ```yaml apiVersion: v1 items: - apiVersion: cluster.redpanda.com/v1alpha1 kind: Topic metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"cluster.redpanda.com/v1alpha1","kind":"Topic","metadata":{"annotations":{},"name":"chat-room","namespace":"redpanda"},"spec":{"additionalConfig":{"cleanup.policy":"compact"},"kafkaApiSpec":{"brokers":["redpanda-0.redpanda.redpanda.svc.cluster.local:9093","redpanda-1.redpanda.redpanda.svc.cluster.local:9093","redpanda-2.redpanda.redpanda.svc.cluster.local:9093"],"tls":{"caCertSecretRef":{"key":"ca.crt","name":"redpanda-default-cert"}}},"partitions":3,"replicationFactor":3}} creationTimestamp: "2023-10-04T10:51:20Z" finalizers: - operator.redpanda.com/finalizer generation: 2 name: chat-room namespace: redpanda resourceVersion: "5262" uid: a26e5a79-cb04-48cb-a2d1-4be9f7f3168f spec: additionalConfig: cleanup.policy: compact interval: 3s kafkaApiSpec: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9093 - redpanda-1.redpanda.redpanda.svc.cluster.local:9093 - redpanda-2.redpanda.redpanda.svc.cluster.local:9093 tls: caCertSecretRef: key: ca.crt name: redpanda-default-cert partitions: 3 replicationFactor: 3 status: conditions: - lastTransitionTime: "2023-10-04T10:54:05Z" message: Topic reconciliation succeeded observedGeneration: 2 reason: Succeeded status: "True" type: Ready observedGeneration: 2 topicConfiguration: - configType: STRING isDefault: false isSensitive: false name: compression.type readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: producer - configType: STRING isDefault: false isSensitive: false name: cleanup.policy readOnly: false source: DYNAMIC_TOPIC_CONFIG unknownTags: {} value: compact - configType: LONG isDefault: false isSensitive: false name: segment.bytes readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "67108864" - configType: LONG isDefault: false isSensitive: false name: retention.ms readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "604800000" - configType: LONG isDefault: false isSensitive: false name: retention.bytes readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "-1" - configType: STRING isDefault: false isSensitive: false name: message.timestamp.type readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: CreateTime - configType: INT isDefault: false isSensitive: false name: max.message.bytes readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "1048576" - configType: BOOLEAN isDefault: false isSensitive: false name: redpanda.remote.read readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "false" - configType: BOOLEAN isDefault: false isSensitive: false name: redpanda.remote.write readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "false" - configType: LONG isDefault: false isSensitive: false name: retention.local.target.bytes readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "-1" - configType: LONG isDefault: false isSensitive: false name: retention.local.target.ms readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "86400000" - configSynonyms: - name: redpanda.remote.delete source: DEFAULT_CONFIG value: "true" configType: BOOLEAN isDefault: false isSensitive: false name: redpanda.remote.delete readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "true" - configType: LONG isDefault: false isSensitive: false name: segment.ms readOnly: false source: DEFAULT_CONFIG unknownTags: {} value: "1209600000" kind: List metadata: resourceVersion: "" ``` To see only the topic configuration as reported by Redpanda: ```bash kubectl get topic --namespace -o go-template=' {{- printf "%-30s %-30s %-30s\n" "KEY" "VALUE" "SOURCE" -}} {{- range .items -}} {{- range .status.topicConfiguration -}} {{- printf "%-30s %-30s %-30s\n" .name .value .source -}} {{- end -}} {{- end -}} ' ``` Example output KEY VALUE SOURCE compression.type producer DEFAULT\_CONFIG cleanup.policy compact DYNAMIC\_TOPIC\_CONFIG segment.bytes 67108864 DEFAULT\_CONFIG retention.ms 604800000 DEFAULT\_CONFIG retention.bytes -1 DEFAULT\_CONFIG message.timestamp.type CreateTime DEFAULT\_CONFIG max.message.bytes 1048576 DEFAULT\_CONFIG redpanda.remote.read false DEFAULT\_CONFIG redpanda.remote.write false DEFAULT\_CONFIG retention.local.target.bytes -1 DEFAULT\_CONFIG retention.local.target.ms 86400000 DEFAULT\_CONFIG redpanda.remote.delete true DEFAULT\_CONFIG segment.ms 1209600000 DEFAULT\_CONFIG ## [](#update-a-topic)Update a topic To update a topic, edit the Topic resource configuration, and apply the changes. > ⚠️ **CAUTION** > > Do not use `rpk` or any other Kafka clients to edit topics that you created using the Topic resource. The topic controller monitors the Redpanda cluster for changes to topics. If you use a Kafka client to edit the topic, the topic controller detects those changes as a drift from the desired state and attempts to reconcile those changes by reverting them to match the Topic resource’s state. The following example changes the cleanup policy for the `chat-room` topic: `example-topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha1 kind: Topic metadata: name: chat-room spec: partitions: 3 replicationFactor: 3 additionalConfig: cleanup.policy: "delete" kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" ``` ```bash kubectl apply -f example-topic.yaml --namespace ``` To set a property back to its default value, remove it from the Redpanda resource. ## [](#delete-a-topic)Delete a topic To delete a topic, delete the Topic resource. For example: ```bash kubectl delete -f example-topic.yaml --namespace ``` When a topic is deleted, its underlying data is deleted, too. > 📝 **NOTE** > > If you delete the Kafka topic directly using a client such as `rpk`, the topic controller recreates an empty topic with the same name. ## [](#delete-records-from-a-topic)Delete records from a topic Redpanda lets you delete data from the beginning of a partition up to a specific event, also known as offset. The offset represents the true creation time of the event, not the time when it was stored by Redpanda. Deleting records frees up disk space, which is especially helpful if your producers are pushing more data than you anticipated in your retention plan. Do this when you know that all consumers have read up to that given offset, and the data is no longer needed. There are different ways to delete records from a topic, including using the [`rpk topic trim-prefix`](https://docs.redpanda.com/streaming/24.1/reference/rpk/rpk-topic/rpk-topic-trim-prefix/) command or using the `DeleteRecords` Kafka API with Kafka clients. > 📝 **NOTE** > > - To delete records, `cleanup.policy` must be set to `delete` or `compact,delete`. > > - Object storage is deleted asynchronously. After messages are deleted, the partition’s start offset will have advanced, but garbage collection of deleted segments may not be complete. > > - Similar to Kafka, after deleting records, local storage and object storage may still contain data for deleted offsets. (Redpanda does not truncate segments. Instead, it bumps the start offset, then it attempts to delete as many whole segments as possible.) Data before the new start offset is not visible to clients but could be read by someone with access to the local disk of a Redpanda node. ## [](#cluster-wide-topic-configurations)Cluster-wide topic configurations Topics provide a way to organize events in a data streaming platform. When you create a topic, the default cluster-level topic configurations are applied using the cluster configuration file, unless you specify different configurations. The following table shows the default cluster-level properties and their equivalent topic-level properties: | Cluster property | Default | Topic property | | --- | --- | --- | | log_cleanup_policy | delete | cleanup.policy | | retention_bytes | null (no limit) | retention.bytes | | log_retention_ms | 604800000 ms (1 week) | retention.ms | | log_segment_ms | null (no limit) | segment.ms | | log_segment_size | 134217728 bytes (128 MiB) | segment.bytes | | log_compression_type | producer | compression.type | | log_message_timestamp_type | CreateTime | message.timestamp.type | | kafka_batch_max_bytes | 1048576 bytes (1 MiB) | max.message.bytes | | write_caching_default | false | write.caching | These default settings are best suited to a one-broker cluster in a development environment. To learn how to modify the default cluster-wide configurations, see [Configure Cluster Properties](https://docs.redpanda.com/streaming/24.1/manage/kubernetes/k-cluster-property-configuration/). Even if you set default values that work for most topics, you may still want to change some properties for a specific topic. > 📝 **NOTE** > > For details about topic properties, see [Topic Configuration Properties](https://docs.redpanda.com/streaming/24.1/reference/properties/topic-properties/). ## [](#suggested-reading)Suggested reading - [Redpanda Topic CRD Reference](https://docs.redpanda.com/streaming/24.1/reference/k-topic-crd/) - [Topic Configuration Properties](https://docs.redpanda.com/streaming/24.1/reference/properties/topic-properties/) ## [](#next-steps)Next steps Combine [SASL authentication](https://docs.redpanda.com/streaming/24.1/manage/kubernetes/security/authentication/k-authentication/) with [authorization](https://docs.redpanda.com/streaming/24.1/manage/security/authorization/) to control which users have permissions to interact with your topics.