Docs Cloud Redpanda Connect Components Catalog Components Catalog Use the following table to search for available inputs, outputs, and processors. Type: All Types Selected ▼ Input Output Processor Scanner Metric Cache Tracer Rate limit Buffer Name Connector Type amqp_0_9 Input, Output archive Processor avro Processor, Scanner aws_bedrock_chat Processor aws_bedrock_embeddings Processor aws_dynamodb Cache, Output aws_dynamodb_partiql Processor aws_kinesis Input, Output aws_kinesis_firehose Output aws_lambda Processor aws_s3 Cache, Input, Output aws_sns Output aws_sqs Input, Output azure_blob_storage Input, Output azure_cosmosdb Input, Output, Processor azure_data_lake_gen2 Output azure_queue_storage Input, Output azure_table_storage Input, Output batched Input benchmark Processor bloblang Processor bounds_check Processor branch Processor broker Input, Output cache Output, Processor cached Processor catch Processor chunker Scanner cohere_chat Processor cohere_embeddings Processor cohere_rerank Processor compress Processor csv Scanner decompress Processor, Scanner dedupe Processor drop Output drop_on Output elasticsearch_v8 Output fallback Output for_each Processor gateway Input gcp_bigquery Output gcp_bigquery_select Input, Processor gcp_cloud_storage Cache, Input, Output gcp_cloudtrace Tracer gcp_pubsub Input, Output gcp_spanner_cdc Input gcp_vertex_ai_chat Processor gcp_vertex_ai_embeddings Processor generate Input git Input google_drive_download Processor google_drive_list_labels Processor google_drive_search Processor group_by Processor group_by_value Processor http Processor http_client Input, Output http_server inproc Input, Output insert_part Processor jmespath Processor jq Processor json_documents Scanner json_schema Processor kafka Input, Output kafka_franz Input, Output lines Scanner local Rate_limit log Processor lru Cache mapping Processor memcached Cache memory Buffer, Cache metric Processor mongodb Cache, Input, Output, Processor mongodb_cdc Input mqtt Input, Output multilevel Cache mutation Processor mysql_cdc Input nats Input, Output nats_jetstream Input, Output nats_kv Cache, Input, Output, Processor nats_request_reply Processor none Buffer, Metric, Tracer noop Cache, Processor openai_chat_completion Processor openai_embeddings Processor openai_image_generation Processor openai_speech Processor openai_transcription Processor openai_translation Processor opensearch Output parallel Processor parquet_decode Processor parquet_encode Processor parse_log Processor pg_stream pinecone Output postgres_cdc Input processors Processor prometheus Metric qdrant Output, Processor questdb Output rate_limit Processor re_match Scanner read_until Input redis Cache, Processor, Rate_limit redis_hash Output redis_list Input, Output redis_pubsub Input, Output redis_scan Input redis_script Processor redis_streams Input, Output redpanda Cache, Input, Output redpanda_common Input, Output redpanda_migrator Input, Output redpanda_migrator_bundle Input, Output redpanda_migrator_offsets Input, Output reject Output reject_errored Output resource Input, Output, Processor retry Output, Processor ristretto Cache schema_registry Input, Output schema_registry_decode Processor schema_registry_encode Processor select_parts Processor sequence Input sftp Input, Output skip_bom Scanner slack Input slack_post Output slack_reaction Output slack_thread Processor slack_users Input sleep Processor snowflake_put Output snowflake_streaming Output spicedb_watch Input split Processor splunk Input splunk_hec Output sql Cache sql_driver_clickhouse sql_driver_mysql sql_driver_oracle sql_driver_postgres sql_driver_sqlite sql_insert Output, Processor sql_raw Input, Output, Processor sql_select Input, Processor switch Output, Processor, Scanner sync_response Output, Processor system_window Buffer tar Scanner text_chunker Processor timeplus Input, Output to_the_end Scanner try Processor ttlru Cache unarchive Processor while Processor workflow Processor xml Processor About Components Every Redpanda Connect pipeline has at least one input, an optional buffer, an output and any number of processors: input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: type: none pipeline: processors: - mapping: | message = this meta.link_count = links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' These are the main components within Redpanda Connect and they provide the majority of useful behavior. Observability components There are also the observability components: logger, metrics, and tracing, which allow you to specify how Redpanda Connect exposes observability data. http: address: 0.0.0.0:4195 enabled: true debug_endpoints: false logger: format: json level: WARN metrics: statsd: address: localhost:8125 flush_period: 100ms tracer: jaeger: agent_address: localhost:6831 Resource components Finally, there are caches and rate limits. These are components that are referenced by core components and can be shared. input: http_client: # This is an input url: TODO rate_limit: foo_ratelimit # This is a reference to a rate limit pipeline: processors: - cache: # This is a processor resource: baz_cache # This is a reference to a cache operator: add key: '${! json("id") }' value: "x" - mapping: root = if errored() { deleted() } rate_limit_resources: - label: foo_ratelimit local: count: 500 interval: 1s cache_resources: - label: baz_cache memcached: addresses: [ localhost:11211 ] It’s also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the resource input, resource output and resource processor respectively. For more information about any of these component types check out their sections: inputs processors outputs buffers metrics tracers logger caches rate limits Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution 🎉 Thanks for your feedback! Unit Testing Inputs