Manage Pipeline Resources on BYOC and Dedicated Clusters

Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance.

Prerequisites

  • A running BYOC (not BYOVPC) or Dedicated cluster.

  • An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the benchmark processor.

Understanding compute units

A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or 100m) and 400 MB (400M) of memory.

For sizing purposes, one compute unit supports an estimated message throughput of 1 MB/s. However, actual performance depends on the complexity of a pipeline, including the components it contains and the processing it does.

You can allocate a maximum of 72 compute units per pipeline. You can add compute units in increments of one up to 15 compute units. Beyond this, scaling options increase to 33 and then to 72 compute units. This scaling strategy is based on the number of machine cores required to provision resources, which scale from two to four, and then to eight cores.

Server resources are charged at an hourly rate in compute unit hours (compute/hour).

Number of compute units CPU Memory

1

0.1 CPU (100m)

400 MB (400M)

2

0.2 CPU (200m)

800 MB (800M)

3

0.3 CPU (300m)

1.2 GB (1200M)

4

0.4 CPU (400m)

1.6 GB (1600M)

5

0.5 CPU (500m)

2.0 GB (2000M)

6

0.6 CPU (600m)

2.4 GB (2400M)

7

0.7 CPU (700m)

2.8 GB (2800M)

8

0.8 CPU (800m)

3.2 GB (3200M)

9

0.9 CPU (900m)

3.6 GB (3600M)

10

1.0 CPU (1000m)

4.0 GB (4000M)

11

1.1 CPU (1100m)

4.4 GB (4400M)

12

1.2 CPU (1200m)

4.8 GB (4800M)

13

1.3 CPU (1300m)

5.2 GB (5200M)

14

1.4 CPU (1400m)

5.6 GB (5600M)

15

1.5 CPU (1500m)

6.0 GB (6000M)

33

3.3 CPU (3300m)

13.2 GB (13200M)

72

7.2 CPU (7200m)

28.8 GB (28800M)
For pipelines with embedded Ollama AI components, one GPU is automatically allocated to the pipeline, which is equivalent to 30 compute units, or 3.0 CPU (3000m) and 12 GB of memory (12000M).

Set an initial resource limit

When you create a data pipeline, you can allocate a fixed amount of server resources to it using compute units.

If your pipeline reaches the CPU limit, it becomes throttled, which reduces the data processing rate. If it reaches the memory limit, the pipeline restarts.

To set an initial resource limit:

  1. Log in to Redpanda Cloud.

  2. On the Clusters page, select the cluster where you want to add a pipeline.

  3. Go to the Connect page.

  4. Select the Redpanda Connect tab.

  5. Click Create pipeline.

  6. Enter details for your pipeline, including a short name and description.

  7. For Compute units, leave the default 1 compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate a maximum of 72 compute units.

  8. For Configuration, paste your pipeline configuration and click Create to run it.

Scale resources

View the server resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption.

To view resources already allocated to a data pipeline:

  • Cloud UI

  • Data Plane API

  1. Log in to Redpanda Cloud.

  2. Go to the cluster where the pipeline is set up.

  3. On the Connect page, select your pipeline and look at the value for Resources.

    • CPU resources are displayed first, in milliCPU. For example, 1 compute unit is 100m or 0.1 CPU.

    • Memory is displayed next in megabytes. For example, 1 compute unit is 400M or 400 MB.

  1. Authenticate and get the base URL for the Data Plane API.

  2. Make a request to GET /v1/redpanda-connect/pipelines, which lists details of all pipelines on your cluster by ID.

    • Memory (memory_shares) is displayed in megabytes. For example, 1 compute unit is 400M or 400 MB.

    • CPU resources (cpu_shares) are displayed milliCPU. For example, 1 compute unit is 100m or 0.1 CPU.

To scale the resources for a pipeline:

  • Cloud UI

  • Data Plane API

  1. Log in to Redpanda Cloud.

  2. Go to the cluster where the pipeline is set up.

  3. On the Connect page, select your pipeline and click Edit.

  4. For Compute units, update the number of compute units. You can allocate a maximum of 72 compute units per pipeline.

  5. Click Update to apply your changes. The specified resources are available immediately.

You can only update CPU resources using the Data Plane API. For every 0.1 CPU that you allocate, Redpanda Cloud automatically reserves 400 MB of memory for the exclusive use of the pipeline.

  1. Authenticate and get the base URL for the Data Plane API, if you haven’t already.

  2. Make a request to GET /v1/redpanda-connect/pipelines/{id}, including the ID of the pipeline you want to update. You’ll use the returned values in the next step.

  3. Now make a request to PUT /v1/redpanda-connect/pipelines/{id}, to update the pipeline resources:

    • Reuse the values returned by your GET request to populate the request body.

    • Replace the cpu_shares value with the resources you want to allocate, and enter any valid value for memory_shares.

      This example allocates 0.2 CPU or 200 milliCPU to a data pipeline. For cpu_shares, 0.1 CPU is the minimum allocation.

      curl -X PUT "https://<data-plane-api-url>/v1/redpanda-connect/pipelines/xxx..." \
 -H 'accept: application/json'\
 -H 'authorization: Bearer xxx...' \
 -H "content-type: application/json" \
 -d '{"config_yaml":"input:\n generate:\n   interval: 1s\n   mapping: |\n     root.id = uuid_v4()\n     root.   user.name = fake(\"name\")\n     root.user.email = fake(\"email\")\n     root.content = fake(\"paragraph\")\n\npipeline:\n processors:\n   - mutation: |\n       root.title = \"PRIVATE AND CONFIDENTIAL\"\n\noutput:\n kafka_franz:\n   seed_brokers:\n     - seed-j888.byoc.prd.cloud.redpanda.com:9092\n   sasl:\n     - mechanism: SCRAM-SHA-256\n       password: password\n       username: connect\n   topic: processed-emails\n   tls:\n     enabled: true\n", \
    "description":"Email processor", \
    "display_name":"emailprocessor-pipeline", \
    "resources":{ \
        "memory_shares":"800M" \
        "cpu_shares":"200m", \
        } \
      }'

      A successful response shows the updated resource allocations with the cpu_shares value returned in milliCPU.

  4. Make a request to GET /v1/redpanda-connect/pipelines to verify your pipeline resource updates.