Docs Connect Introduction to Redpanda Connect Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction based resiliency with back pressure, so when connecting to at-least-once sources and sinks it’s able to guarantee at-least-once delivery without needing to persist messages during transit. It’s simple to deploy, comes with a wide range of connectors, and is totally data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative. Get started. Components Inputs Services AWS Azure Utility Integration Local Social GCP Network Inputs that consume from storage or message streaming services. amqp_0_9 amqp_1 AWS Kinesis AWS S3 AWS SQS Azure Blob Storage Azure Queue Storage Azure Table Storage beanstalkd Cassandra Discord gcp_bigquery_select GCP Cloud Storage GCP Pub/Sub HDFS Kafka Franz-go MongoDB MQTT NATS nats_jetstream nats_kv nats_stream NSQ Pulsar redis_list redis_pubsub redis_scan redis_streams splunk sql_raw sql_select X/Twitter Inputs that consume from Amazon Web Services products. AWS Kinesis AWS S3 AWS SQS Inputs that consume from Microsoft Azure services. Azure Blob Storage Azure Cosmos DB Azure Queue Storage Azure Table Storage Inputs that provide utility by generating data or combining/wrapping other inputs. batched broker dynamic generate inproc read_until resource sequence subprocess CockroachDB Schema Registry Inputs that consume from the local machine/filesystem. CSV file Parquet stdin Inputs that consume from social applications and services. Discord X/Twitter Inputs that consume from Google Cloud Platform services. gcp_bigquery_select GCP Cloud Storage GCP Pub/Sub Inputs that consume directly from low level network protocols. http_client http_server nanomsg sftp Socket socket_server WebSocket zmq4 Processors Parsing Utility Mapping AI Integration Azure Composition Services Processors that specialize in translating messages from one format to another. archive Avro bloblang compress decompress Grok mapping MessagePack mutation parquet_decode parquet_encode parse_log Protobuf schema_registry_decode schema_registry_encode unarchive XML Processors that provide general utility or do not fit in another category. archive bounds_check cached dedupe log metric rate_limit redpanda_data_transform resource select_parts sleep split sync_response unarchive WebAssembly Processors that specialize in restructuring messages. AWK bloblang jq mapping JSON Schema JMESPath JavaScript mutation aws_bedrock_chat gcp_vertex_ai_chat ollama_chat ollama_embeddings openai_chat_completion openai_embeddings openai_image_generation openai_speech openai_transcription openai_translation Processors that interact with external services. aws_dynamodb_partiql AWS Lambda cache command Couchbase gcp_bigquery_select HTTP Redis redis_script schema_registry_decode schema_registry_encode sql_insert sql_raw sql_select subprocess Azure Cosmos DB Higher level processors that compose other processors and modify their behavior. branch catch for_each group_by group_by_value insert_part parallel processors retry switch try while workflow MongoDB nats_kv nats_request_reply Outputs Services AWS Azure Utility Social Local GCP Network AI Integration Outputs that write to storage or message streaming services. amqp_0_9 amqp_1 AWS DynamoDB AWS Kinesis AWS Kinesis Firehose AWS S3 AWS SNS AWS SQS Azure Blob Storage Azure Queue Storage Azure Table Storage beanstalkd cache Discord Elasticsearch GCP BigQuery GCP Cloud Storage GCP Pub/Sub HDFS Kafka Franz-go MongoDB MQTT NATS nats_jetstream nats_kv nats_stream NSQ OpenSearch Pulsar pusher redis_hash redis_list redis_pubsub redis_streams Snowflake Splunk sql_insert sql_raw Outputs that write to Amazon Web Services products. AWS DynamoDB AWS Kinesis AWS Kinesis Firehose AWS S3 AWS SNS AWS SQS Outputs that write to Microsoft Azure services. Azure Blob Storage Azure Cosmos DB Azure Queue Storage Azure Table Storage Outputs that provide utility by combining/wrapping other outputs. broker drop drop_on dynamic fallback inproc reject reject_errored resource retry subprocess switch sync_response Outputs that write to social applications and services. Discord Outputs that write to the local machine/filesystem. file stdout Outputs that write to Google Cloud Platform services. GCP BigQuery GCP Cloud Storage GCP Pub/Sub Outputs that write directly to low level network protocols. http_client http_server nanomsg sftp Socket WebSocket zmq4 pinecone qdrant Schema Registry Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Configuration