Docs Cloud Redpanda Connect Components Catalog Components Catalog Page options Copy as Markdown Copied! View as plain text Ask AI about this topic Add MCP server to VS Code Use the following table to search for available inputs, outputs, and processors. Type: All Types Selected ▼ Processor Input Output Scanner Metric Cache Tracer Rate limit Buffer Name Connector Type a2a_message amqp_0_9RabbitMQ AMQP Input, Output archiveZIP TAR GZIP Processor avro Processor, Scanner aws_bedrock_chatAmazon AWS Bedrock Chat Processor aws_bedrock_embeddingsAmazon AWS Bedrock Embeddings Processor aws_dynamodbAWS DynamoDB Amazon DynamoDB DynamoDB Cache, Output aws_dynamodb_partiqlAmazon AWS DynamoDB PartiQL Processor aws_kinesisAWS Kinesis Amazon Kinesis Kinesis Input, Output aws_kinesis_firehoseAWS Kinesis Firehose Amazon Kinesis Firehose Kinesis Firehose Output aws_lambdaAWS Lambda Amazon Lambda Lambda Processor aws_s3AWS S3 Amazon S3 S3 Simple Storage Service Cache, Input, Output aws_snsAWS SNS Amazon SNS SNS Simple Notification Service Output aws_sqsAWS SQS Amazon SQS SQS Simple Queue Service Input, Output azure_blob_storageAzure Blob Storage Microsoft Azure Storage Input, Output azure_cosmosdbMicrosoft Azure Azure Input, Output, Processor azure_data_lake_gen2Microsoft Azure Azure Output azure_queue_storageAzure Queue Storage Microsoft Azure Queue Input, Output azure_table_storageAzure Table Storage Microsoft Azure Table Input, Output batched Input benchmark Processor bloblang Processor bounds_check Processor branch Processor broker Input, Output cache Output, Processor cached Processor catch Processor chunker Scanner cohere_chat Processor cohere_embeddings Processor cohere_rerank Processor compress Processor csvComma-Separated Values Scanner cyborgdb Output decompress Processor, Scanner dedupe Processor drop Output drop_on Output elasticsearch_v8 Output fallback Output for_each Processor gateway Input gcp_bigqueryGCP BigQuery Google BigQuery BigQuery Output gcp_bigquery_selectGCP BigQuery Google Cloud GCP Input, Processor gcp_cloud_storageGCP Cloud Storage Google Cloud Storage GCS Cache, Input, Output gcp_cloudtraceGCP Cloud Trace Tracer gcp_pubsubGCP PubSub Google Cloud Pub/Sub GCP Pub/Sub Google Pub/Sub Input, Output gcp_spanner_cdcGoogle Cloud GCP Input gcp_vertex_ai_chatGCP Vertex AI Google Cloud GCP Processor gcp_vertex_ai_embeddingsGoogle Cloud GCP Processor generate Input git Input google_drive_download Processor google_drive_list_labels Processor google_drive_search Processor group_by Processor group_by_value Processor http Processor http_clientHTTP REST API REST Input, Output http_serverHTTP REST API REST Gateway inproc Input, Output insert_part Processor jira Processor jmespath Processor jq Processor json_array Scanner json_documents Scanner json_schemaJSON Schema Processor kafkaApache Kafka Input, Output kafka_franzApache Kafka Kafka Input, Output legacy_redpanda_migratorredpanda_migrator Input, Output legacy_redpanda_migrator_offsetsredpanda_migrator_offsets Input, Output lines Scanner local Rate_limit log Processor lru Cache mapping Processor memcached Cache memory Buffer, Cache metric Processor microsoft_sql_server_cdc Input mongodbMongo Cache, Input, Output, Processor mongodb_cdcMongoDB CDC Input mqtt Input, Output multilevel Cache mutation Processor mysql_cdc Input natsNATS.io Input, Output nats_jetstreamNATS JetStream NATS Input, Output nats_kvNATS KV Cache, Input, Output, Processor nats_request_replyNATS Request Reply Processor none Buffer, Metric, Tracer noop Cache, Processor openai_chat_completion Processor openai_embeddings Processor openai_image_generation Processor openai_speech Processor openai_transcription Processor openai_translation Processor opensearch Output otlp_grpcOpenTelemetry OTLP OTel gRPC Input, Output otlp_httpOpenTelemetry OTLP OTel Input, Output parallel Processor parquet_decode Processor parquet_encode Processor parse_log Processor pg_stream pinecone Output postgres_cdc Input processors Processor prometheus Metric qdrant Output, Processor questdb Output rate_limit Processor re_match Scanner read_until Input redis Cache, Processor, Rate_limit redis_hashRedis Hash Redis Output redis_listRedis List Redis Lists Redis Input, Output redis_pubsubRedis PubSub Redis Pub/Sub Redis Input, Output redis_scanRedis Input redis_scriptRedis Script Processor redis_streamsRedis Streams Redis Input, Output redpanda Cache, Input, Output, Tracer redpanda_common Input, Output redpanda_migrator Input, Output redpanda_migrator_bundle Input, Output reject Output reject_errored Output resource Input, Output, Processor retry Output, Processor ristretto Cache schema_registry Input, Output schema_registry_decode Processor schema_registry_encode Processor select_parts Processor sequence Input sftp Input, Output skip_bom Scanner slack Input slack_postSlack Post Output slack_reactionSlack Reaction Output slack_threadSlack Thread Processor slack_usersSlack Users Input sleep Processor snowflake_putSnowflake Output snowflake_streamingSnowflake Streaming Output spicedb_watch Input split Processor splunk Input splunk_hecSplunk Output sql Cache sql_driver_clickhouseClickHouse sql_driver_mysqlMYSQL sql_driver_oracleOracle sql_driver_postgresPostgreSQL sql_driver_sqliteSQLite sql_insertSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino Output, Processor sql_rawSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino Input, Output, Processor sql_selectSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino Input, Processor switch Output, Processor, Scanner sync_response Output, Processor system_window Buffer tar Scanner text_chunker Processor timeplus Input, Output to_the_end Scanner try Processor ttlru Cache unarchiveZIP TAR GZIP Archive Processor while Processor workflow Processor xml Processor About Components Every Redpanda Connect pipeline has at least one input, an optional buffer, an output and any number of processors: input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: type: none pipeline: processors: - mapping: | message = this meta.link_count = links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' These are the main components within Redpanda Connect and they provide the majority of useful behavior. Observability components There are also the observability components: logger, metrics, and tracing, which allow you to specify how Redpanda Connect exposes observability data. http: address: 0.0.0.0:4195 enabled: true debug_endpoints: false logger: format: json level: WARN metrics: statsd: address: localhost:8125 flush_period: 100ms tracer: jaeger: agent_address: localhost:6831 Resource components Finally, there are caches and rate limits. These are components that are referenced by core components and can be shared. input: http_client: # This is an input url: TODO rate_limit: foo_ratelimit # This is a reference to a rate limit pipeline: processors: - cache: # This is a processor resource: baz_cache # This is a reference to a cache operator: add key: '${! json("id") }' value: "x" - mapping: root = if errored() { deleted() } rate_limit_resources: - label: foo_ratelimit local: count: 500 interval: 1s cache_resources: - label: baz_cache memcached: addresses: [ localhost:11211 ] It’s also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the resource input, resource output and resource processor respectively. For more information about any of these component types check out their sections: inputs processors outputs buffers metrics tracers logger caches rate limits Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution 🎉 Thanks for your feedback! Unit Testing Inputs