# Manage Pipeline Resources on Clusters > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Pipeline Resources on Clusters latest-operator-version: v26.1.4 latest-console-tag: v3.7.3 latest-connect-version: 4.93.0 latest-redpanda-tag: v26.1.9 docname: connect/configuration/resource-management page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/resource-management.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/resource-management.adoc description: Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. page-git-created-date: "2024-12-18" page-git-modified-date: "2026-05-26" --- Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. ## [](#prerequisites)Prerequisites - A running Redpanda Cloud cluster. - An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the [`benchmark` processor](https://docs.redpanda.com/connect/components/processors/benchmark/). ### [](#understanding-compute-units)Understanding compute units A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory. For sizing purposes, one compute unit supports an estimated message throughput of 1 MB/s. However, actual performance depends on the complexity of a pipeline, including the components it contains and the processing it does. You can allocate a maximum of 72 compute units per pipeline. You can add compute units in increments of one up to 15 compute units. Beyond this, scaling options increase to 33 and then to 72 compute units. This scaling strategy is based on the number of machine cores required to provision resources, which scale from two to four, and then to eight cores. Server resources are charged at an [hourly rate in compute unit hours (compute/hour)](https://docs.redpanda.com/cloud-data-platform/billing/billing/#redpanda-connect-pipeline-metrics). | Number of compute units | CPU | Memory | | --- | --- | --- | | 1 | 0.1 CPU (100m) | 400 MB (400M) | | 2 | 0.2 CPU (200m) | 800 MB (800M) | | 3 | 0.3 CPU (300m) | 1.2 GB (1200M) | | 4 | 0.4 CPU (400m) | 1.6 GB (1600M) | | 5 | 0.5 CPU (500m) | 2.0 GB (2000M) | | 6 | 0.6 CPU (600m) | 2.4 GB (2400M) | | 7 | 0.7 CPU (700m) | 2.8 GB (2800M) | | 8 | 0.8 CPU (800m) | 3.2 GB (3200M) | | 9 | 0.9 CPU (900m) | 3.6 GB (3600M) | | 10 | 1.0 CPU (1000m) | 4.0 GB (4000M) | | 11 | 1.1 CPU (1100m) | 4.4 GB (4400M) | | 12 | 1.2 CPU (1200m) | 4.8 GB (4800M) | | 13 | 1.3 CPU (1300m) | 5.2 GB (5200M) | | 14 | 1.4 CPU (1400m) | 5.6 GB (5600M) | | 15 | 1.5 CPU (1500m) | 6.0 GB (6000M) | | 33 | 3.3 CPU (3300m) | 13.2 GB (13200M) | | 72 | 7.2 CPU (7200m) | 28.8 GB (28800M) | > 📝 **NOTE** > > A GPU machine is automatically assigned to each pipeline that contains embedded Ollama AI components. By default, GPU-enabled pipelines are allocated eight compute units. For larger workloads, you can scale them up to a maximum of 30 compute units. ### [](#set-an-initial-resource-limit)Set an initial resource limit When you create a data pipeline, you can allocate a fixed amount of server resources to it using compute units. > 📝 **NOTE** > > If your pipeline reaches the CPU limit, it becomes throttled, which reduces the data processing rate. If it reaches the memory limit, the pipeline restarts. To set an initial resource limit: 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. On the **Clusters** page, select the cluster where you want to add a pipeline. 3. Go to the **Connect** page. 4. Select the **Redpanda Connect** tab. 5. Click **Create pipeline**. 6. Enter details for your pipeline, including a short name and description. 7. For **Compute units**, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate a maximum of 72 compute units. 8. For **Configuration**, paste your pipeline configuration and click **Create** to run it. ### [](#scale-resources)Scale resources View the server resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption. To view resources already allocated to a data pipeline: #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the cluster where the pipeline is set up. 3. On the **Connect** page, select your pipeline and look at the value for **Resources**. - CPU resources are displayed first, in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU. - Memory is displayed next in megabytes. For example, `1` compute unit is `400M` or 400 MB. #### Data Plane API 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API. 2. Make a request to [`GET /v1/redpanda-connect/pipelines`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_listpipelines), which lists details of all pipelines on your cluster by ID. - Memory (`memory_shares`) is displayed in megabytes. For example, `1` compute unit is `400M` or 400 MB. - CPU resources (`cpu_shares`) are displayed in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU. To scale the resources for a pipeline: #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the cluster where the pipeline is set up. 3. On the **Connect** page, select your pipeline and click **Edit**. 4. For **Compute units**, update the number of compute units. You can allocate a maximum of 72 compute units per pipeline. 5. Click **Update** to apply your changes. The specified resources are available immediately. #### Data Plane API You can only update CPU resources using the Data Plane API. For every 0.1 CPU that you allocate, Redpanda Cloud automatically reserves 400 MB of memory for the exclusive use of the pipeline. 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API, if you haven’t already. 2. Make a request to [`GET /v1/redpanda-connect/pipelines/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_getpipeline), including the ID of the pipeline you want to update. You’ll use the returned values in the next step. 3. Now make a request to [`PUT /v1/redpanda-connect/pipelines/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_updatepipeline), to update the pipeline resources: - Reuse the values returned by your `GET` request to populate the request body. - Replace the `cpu_shares` value with the resources you want to allocate, and enter any valid value for `memory_shares`. This example allocates 0.2 CPU or 200 milliCPU to a data pipeline. For `cpu_shares`, `0.1` CPU is the minimum allocation. ```bash curl -X PUT "https:///v1/redpanda-connect/pipelines/xxx..." \ -H 'accept: application/json'\ -H 'authorization: Bearer xxx...' \ -H "content-type: application/json" \ -d '{ "config_yaml": "input:\n generate:\n interval: 1s\n mapping: |\n root.id = uuid_v4()\n root.user.name = fake(\"name\")\n root.user.email = fake(\"email\")\n root.content = fake(\"paragraph\")\n\npipeline:\n processors:\n - mutation: |\n root.title = \"PRIVATE AND CONFIDENTIAL\"\n\noutput:\n kafka_franz:\n seed_brokers:\n - seed-j888.byoc.prd.cloud.redpanda.com:9092\n sasl:\n mechanism: SCRAM-SHA-256\n password: password\n username: connect\n topic: processed-emails\n tls:\n enabled: true\n", "description": "Email processor", "display_name": "emailprocessor-pipeline", "resources": { "memory_shares": "800M", "cpu_shares": "200m" } }' ``` A successful response shows the updated resource allocations with the `cpu_shares` value returned in milliCPU. 4. Make a request to [`GET /v1/redpanda-connect/pipelines`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_listpipelines) to verify your pipeline resource updates.