Sizing Connectors
Connector tasks
When you set up a connector, its main responsibility is to validate the configuration and spawn connector tasks, which perform the work. Setting up multiple tasks for a connector allows for parallelization of the work, resulting in higher throughputs.
Before setting up connector tasks, consider the following:
-
For source connectors, the ability to add tasks to achieve higher throughput depends on the connector implementation and configuration. For many connectors, only a single connector task is allowed (for example, Debezium allows a single task only). When Redpanda Cloud does not offer an option to set the number of tasks, the source connector runs only one task.
-
For sink connectors, parallelism is achieved by evenly distributing configured topic partitions for the connector amongst connector tasks. The number of partitions must be equal to or greater than the number of tasks.
Single task throughput
Connector throughput depends on many factors, including converters used, compression, message size, and the performance of external systems. As a rule of thumb, expect a single connector task to provide 1-2 MB/s of throughput.
Specify number of connector tasks for a sink connector
It can be a challenge to determine the number of connector tasks to use for a given workload, so you must experiment to find the right number. Start with low number of connector tasks and wait a couple of minutes to view performance. Keep increasing the number of tasks until satisfactory throughput is achieved. Keep in mind that the underlying infrastructure must scale to provide room for additional connector tasks. Waiting roughly 10 minutes after each change should provide sufficient time for the system to scale up.