# Autoscale Redpanda Connect Pipelines > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Autoscale Redpanda Connect Pipelines page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-redpanda-tag: v26.2.1 latest-console-tag: v3.10.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.105.0 docname: kubernetes/k-autoscale-connect-pipelines page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: kubernetes/k-autoscale-connect-pipelines.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc description: Scale Redpanda Connect pipelines manually or automatically with kubectl scale, a HorizontalPodAutoscaler, or KEDA. page-topic-type: how-to # Beta release status page-beta: "true" personas: platform_operator, platform_engineer learning-objective-1: Scale a pipeline manually through the scale subresource learning-objective-2: Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler learning-objective-3: Autoscale a pipeline on consumer group lag or Connect metrics with KEDA page-git-created-date: "2026-07-28" page-git-modified-date: "2026-07-28" support-status: supported release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Match pipeline capacity to your workload by scaling the number of pipeline replicas, either manually or automatically based on resource usage, consumer group lag, or the metrics that Redpanda Connect emits. After reading this page, you will be able to: - Scale a pipeline manually through the scale subresource - Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler - Autoscale a pipeline on consumer group lag or Connect metrics with KEDA This feature is in beta and requires Redpanda Operator 26.2 or later. Beta features are not recommended for production environments. The [Pipeline resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-connect-pipelines/) implements the Kubernetes [scale subresource](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), so anything that speaks the `/scale` API can drive `spec.replicas`: `kubectl scale`, a HorizontalPodAutoscaler (HPA), or a [KEDA](https://keda.sh/) ScaledObject. > ❗ **IMPORTANT** > > Always target the Pipeline resource, never its Deployment. The operator resets the Deployment’s replica count to the Pipeline’s value on every sync, so an autoscaler pointed at the Deployment is continuously fought and undone. ## [](#prerequisites)Prerequisites You must have the following: - A running pipeline managed by the Redpanda Operator with the Connect controller enabled. See [Run Redpanda Connect Pipelines in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-connect-pipelines/). - For HPA on CPU or memory: [metrics-server](https://github.com/kubernetes-sigs/metrics-server) installed in the cluster. - For KEDA: [KEDA](https://keda.sh/docs/latest/deploy/) installed in the cluster. - For scaling on Redpanda Connect metrics: a Prometheus installation that scrapes pipeline Pods, for example [kube-prometheus-stack](https://github.com/prometheus-operator/kube-prometheus) together with the operator chart’s `connectController.monitoring.enabled=true` setting. Configure only one autoscaler per pipeline. An HPA and a KEDA ScaledObject targeting the same Pipeline fight each other. ## [](#scale-a-pipeline-manually)Scale a pipeline manually Scale the Pipeline directly with `kubectl scale`: ```bash kubectl scale pipeline/ --replicas=3 --namespace ``` The operator updates the pipeline’s Deployment to match, and the pipeline’s consumer group rebalances its partitions across the new replicas. To confirm that the scale subresource is wired up, read it directly: ```bash kubectl get --raw "/apis/cluster.redpanda.com/v1alpha2/namespaces//pipelines//scale" ``` The response reports `spec.replicas`, `status.replicas`, and `status.selector`, the label selector that autoscalers use to find the pipeline’s Pods. To list a pipeline’s Pods yourself, the `app.kubernetes.io/instance=` label is the simplest filter. > 📝 **NOTE** > > When you scale a pipeline that consumes from Redpanda, replicas beyond the topic’s partition count sit idle, because a consumer group assigns each partition to at most one member. Size `--replicas`, and any autoscaler’s maximum, to the partition count of the input topic. ## [](#autoscale-on-cpu-and-memory-with-hpa)Autoscale on CPU and memory with HPA CPU-based and memory-based HPAs work without any extra configuration, because pipeline Pods always carry resource requests: if you don’t set `spec.resources`, the operator applies default requests of `100m` CPU and `256Mi` memory. 1. Ensure metrics-server is running. If `kubectl top pods` returns metrics, you’re ready. Otherwise, install it: ```bash kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml ``` 2. Set explicit resource requests on the pipeline, so utilization percentages are meaningful for your workload: ```yaml spec: resources: requests: cpu: 100m memory: 256Mi limits: cpu: 300m memory: 512Mi ``` 3. Create an HPA whose `scaleTargetRef` points at the Pipeline: `pipeline-hpa.yaml` ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: orders-to-warehouse namespace: spec: scaleTargetRef: apiVersion: cluster.redpanda.com/v1alpha2 kind: Pipeline name: orders-to-warehouse minReplicas: 1 maxReplicas: 4 behavior: scaleUp: stabilizationWindowSeconds: 0 scaleDown: stabilizationWindowSeconds: 60 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 80 ``` 4. Apply the manifest and watch the HPA react. Resource metrics lag by about a minute: ```bash kubectl apply -f pipeline-hpa.yaml kubectl get hpa orders-to-warehouse --namespace --watch ``` Example output while the pipeline is busy. CPU is at 300% of its request, above the 50% target, so the HPA scales out toward `maxReplicas`: NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE orders-to-warehouse Pipeline/orders-to-warehouse cpu: 300%/50%, memory: 30%/80% 1 4 4 2m 5. Verify that the Pipeline followed: ```bash kubectl get pipeline orders-to-warehouse --namespace kubectl get pods -l app.kubernetes.io/instance=orders-to-warehouse --namespace ``` To scale an HPA on the metrics Redpanda Connect emits instead of CPU or memory, either expose them to the HPA as custom metrics through [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter), or use KEDA’s Prometheus trigger, described next, which queries Prometheus directly and needs no adapter. ## [](#autoscale-with-keda)Autoscale with KEDA KEDA drives the same scale subresource, but adds event-driven triggers, such as consumer group lag and arbitrary Prometheus queries, and can scale a pipeline to zero when it’s idle. Under the hood, KEDA materializes and manages an HPA named `keda-hpa-` for you. Install KEDA if it isn’t already present: ```bash helm repo add kedacore https://kedacore.github.io/charts helm upgrade --install keda kedacore/keda --namespace keda --create-namespace ``` ### [](#scale-on-consumer-group-lag)Scale on consumer group lag For pipelines that consume from Redpanda, consumer group lag is usually the signal you actually want: it measures how far the pipeline is behind, not how busy its CPUs are. KEDA’s `kafka` trigger works with Redpanda because Redpanda is Kafka API-compatible. 1. Create a ScaledObject that targets the Pipeline and watches the pipeline’s consumer group: `scaledobject-lag.yaml` ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: orders-to-warehouse namespace: spec: scaleTargetRef: apiVersion: cluster.redpanda.com/v1alpha2 kind: Pipeline name: orders-to-warehouse minReplicaCount: 1 maxReplicaCount: 6 pollingInterval: 10 # How often KEDA evaluates the trigger (seconds). cooldownPeriod: 120 # Delay before scaling to zero (only with minReplicaCount: 0). advanced: horizontalPodAutoscalerConfig: behavior: scaleUp: stabilizationWindowSeconds: 0 scaleDown: stabilizationWindowSeconds: 60 policies: - type: Pods value: 2 periodSeconds: 30 triggers: - type: kafka metadata: bootstrapServers: ..svc.cluster.local:9093 consumerGroup: orders-to-warehouse-ingest topic: orders lagThreshold: "500" # Target lag per replica. offsetResetPolicy: earliest allowIdleConsumers: "false" # Never scale beyond the partition count. ``` Replace `..svc.cluster.local:9093` with the bootstrap address of your cluster’s internal Kafka listener, and set `consumerGroup` and `topic` to match the pipeline’s `input.redpanda` block. KEDA sizes the pipeline at roughly `total lag / lagThreshold` replicas, clamped between `minReplicaCount` and `maxReplicaCount`. With `allowIdleConsumers: "false"`, KEDA also caps replicas at the topic’s partition count, matching consumer group parallelism. 2. If the listener requires TLS or SASL, add a [TriggerAuthentication](https://keda.sh/docs/latest/scalers/apache-kafka/) and reference it from the trigger: ```yaml apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: redpanda-kafka-auth namespace: spec: secretTargetRef: - parameter: sasl name: keda-redpanda-auth key: sasl # For example: scram_sha512 - parameter: username name: keda-redpanda-auth key: username - parameter: password name: keda-redpanda-auth key: password - parameter: tls name: keda-redpanda-auth key: tls # enable - parameter: ca name: keda-redpanda-auth key: ca.crt ``` Reference it in the ScaledObject trigger with `authenticationRef.name: redpanda-kafka-auth`. The referenced user needs `Describe` permissions on the topic and consumer group to read lag. 3. Apply the manifest and watch the pipeline scale as lag grows and drains: ```bash kubectl apply -f scaledobject-lag.yaml kubectl get scaledobject orders-to-warehouse --namespace kubectl get pipeline orders-to-warehouse --namespace --watch ``` As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`. The consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. ### [](#scale-on-redpanda-connect-metrics)Scale on Redpanda Connect metrics Redpanda Connect emits pipeline-level metrics such as `input_received`, `output_sent`, and latency timings on each Pod’s `http` port (4195) at `/metrics`. Scaling on these lets you react to actual pipeline throughput even when the input isn’t a Redpanda topic. 1. Get the metrics into Prometheus. With Prometheus Operator installed, set `connectController.monitoring.enabled=true` on the operator chart so it renders a PodMonitor for every pipeline: ```bash helm upgrade --install redpanda-operator redpanda/operator \ --namespace \ --reuse-values \ --set connectController.monitoring.enabled=true kubectl get podmonitor --namespace ``` 2. Create a ScaledObject with a `prometheus` trigger. This example adds one replica for every 5,000 messages per second the pipeline receives: `scaledobject-metrics.yaml` ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: orders-to-warehouse-throughput namespace: spec: scaleTargetRef: apiVersion: cluster.redpanda.com/v1alpha2 kind: Pipeline name: orders-to-warehouse minReplicaCount: 1 maxReplicaCount: 4 triggers: - type: prometheus metadata: serverAddress: http://prometheus-operated.monitoring.svc:9090 query: sum(rate(input_received{namespace="", pod=~"orders-to-warehouse-.*"}[2m])) threshold: "5000" ``` 3. Apply the manifest and confirm that KEDA created its HPA against the Pipeline: ```bash kubectl apply -f scaledobject-metrics.yaml kubectl get hpa keda-hpa-orders-to-warehouse-throughput --namespace ``` ### [](#scale-to-zero)Scale to zero Set `minReplicaCount: 0` on a ScaledObject to let KEDA stop an idle pipeline entirely and restart it when the trigger fires again, for example when lag reappears on the consumer group. KEDA manages the 0 to 1 transition itself, waiting `cooldownPeriod` seconds after the last trigger activity before scaling to zero. A plain HPA cannot do this. ## [](#how-autoscaling-interacts-with-the-pipeline-spec)How autoscaling interacts with the Pipeline spec - **`spec.paused` wins over the autoscaler.** A paused pipeline stays at zero replicas (phase `Stopped`) even while an HPA or KEDA keeps writing `spec.replicas`. When you unpause it, the pipeline resumes at the autoscaler-managed count. - **An explicit `spec.replicas: 0` is preserved.** The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0. This is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. - **Stop declaring `replicas` in manifests once an autoscaler owns it.** Every re-apply of a manifest that sets `spec.replicas` overwrites the autoscaler’s value. Remove the field from the manifests you apply, or configure your GitOps tool to ignore it, for example with Argo CD’s `ignoreDifferences`. ## [](#next-steps)Next steps - [Run Redpanda Connect Pipelines in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-connect-pipelines/) - [KEDA scalers reference](https://keda.sh/docs/latest/scalers/) - [Kubernetes HPA documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs)